squirrel.fsspec.custom_gcsfs

Module Contents

Classes

CustomGCSFileSystem

Monkey patch GCSFileSystem with a custom_retry() function that adds http error code 400 to be retriable.

Functions

custom_retry(→ bool)

We have intermittent failures in reading data and on retry it works, it could be because of this issue

class squirrel.fsspec.custom_gcsfs.CustomGCSFileSystem(token: str = 'google_default', access: str = 'read_write', **kwargs)

Bases: gcsfs.GCSFileSystem

Monkey patch GCSFileSystem with a custom_retry() function that adds http error code 400 to be retriable. Other error codes are already included in gcsfs.utils.is_retriable().

Overwrite gcsfs.utils.is_retriable

squirrel.fsspec.custom_gcsfs.custom_retry(exception: Exception) → bool

We have intermittent failures in reading data and on retry it works, it could be because of this issue https://github.com/dask/gcsfs/issues/290 hence monkey patching to include 400 as retriable other relevant issues: https://github.com/dask/gcsfs/issues/316, https://github.com/dask/gcsfs/issues/327, https://github.com/dask/gcsfs/issues/323