squirrel.fsspec.custom_gcsfs
¶
Module Contents¶
Classes¶
Monkey patch GCSFileSystem with a |
Functions¶
|
We have intermittent failures in reading data and on retry it works, it could be because of this issue |
-
class
squirrel.fsspec.custom_gcsfs.
CustomGCSFileSystem
(token: str = 'google_default', access: str = 'read_write', **kwargs)¶ Bases:
gcsfs.GCSFileSystem
Monkey patch GCSFileSystem with a
custom_retry()
function that adds http error code 400 to be retriable. Other error codes are already included ingcsfs.utils.is_retriable()
.Overwrite gcsfs.utils.is_retriable
-
squirrel.fsspec.custom_gcsfs.
custom_retry
(exception: Exception) → bool¶ We have intermittent failures in reading data and on retry it works, it could be because of this issue https://github.com/dask/gcsfs/issues/290 hence monkey patching to include 400 as retriable other relevant issues: https://github.com/dask/gcsfs/issues/316, https://github.com/dask/gcsfs/issues/327, https://github.com/dask/gcsfs/issues/323