squirrel.store.filesystem

Module Contents

Classes

FilesystemStore

Store that uses fsspec to read from / write to files.

Functions

get_random_key(→ str)

Generates a random key
class squirrel.store.filesystem.FilesystemStore(url: str, serializer: Optional[squirrel.serialization.SquirrelSerializer] = None, clean: bool = False, **storage_options)

Bases: squirrel.store.store.AbstractStore

Store that uses fsspec to read from / write to files.

Initializes FilesystemStore.

Parameters

  • url (str) – Path to the root directory. If this path does not exist, it will be created.

  • serializer (SquirrelSerializer, optional) – Serializer that is used to serialize data before persisting (see set()) and to deserialize data after reading (see get()). If not specified, data will not be (de)serialized. Defaults to None.

  • clean (bool) – If true, all files in the store will be removed recursively

  • **storage_options – Keyword arguments passed to filesystem initializer.

get(key: str, mode: str = 'rb', **open_kwargs) → Any

Yields the item with the given key.

If the store has a serializer, data read from the file will be deserialized.

Parameters

  • key (str) – Key corresponding to the item to retrieve.

  • mode (str) – IO mode to use when opening the file. Defaults to “rb”.

  • **open_kwargs – Keyword arguments that will be forwarded to the filesystem object when opening the file.

Yields

(Any) Item with the given key.

keys(nested: bool = True, **kwargs) → Iterator[str]

Yields all paths in the store, relative to the root directory.

Paths are generated using squirrel.iterstream.source.FilePathGenerator.

Parameters

  • nested (bool) – Whether to return paths that are not direct children of the root directory. If True, all paths in the store will be yielded. Otherwise, only the top-level paths (i.e. direct children of the root path) will be yielded. This option is passed to FilePathGenerator initializer. Defaults to True.

  • **kwargs – Other keyword arguments passed to the FilePathGenerator initializer. If a key is present in both kwargs and self.storage_options, the value from kwargs will be used.

Yields

(str) Paths to files and directories in the store relative to the root directory.

set(value: Any, key: Optional[str] = None, mode: str = 'wb', **open_kwargs) → None

Persists an item with the given key.

If the store has a serializer, data item will be serialized before writing to a file.

Parameters

  • value (Any) – Item to be persisted.

  • key (Optional[str]) – Optional key corresponding to the item to persist.

  • mode (str) – IO mode to use when opening the file. Defaults to “wb”.

  • **open_kwargs – Keyword arguments that will be forwarded to the filesystem object when opening the file.

squirrel.store.filesystem.get_random_key(length: int = 16) → str

Generates a random key