squirrel.artifact_manager.filesystem

Module Contents

Classes

ArtifactFileStore	A FilesystemStore serving as the backend for the FileSystemArtifactManager.
FileSystemArtifactManager	Helper class that provides a standard way to create an ABC using

Attributes

Serializers

squirrel.artifact_manager.filesystem.Serializers

class squirrel.artifact_manager.filesystem.ArtifactFileStore(url: str, serializer: Optional[squirrel.serialization.SquirrelSerializer] = None, clean: bool = False, **storage_options)

Bases: squirrel.store.FilesystemStore

A FilesystemStore serving as the backend for the FileSystemArtifactManager.

The get and set methods are altered to allow for storing serialized data as well as raw files.

If the final path component is a serializer name, the data is stored as a serialized file. If the final path component is “files”, the data is stored as a raw file.

Initializes FilesystemStore.

Parameters

• url (str) – Path to the root directory. If this path does not exist, it will be created.

• serializer (SquirrelSerializer, optional) – Serializer that is used to serialize data before persisting (see set()) and to deserialize data after reading (see get()). If not specified, data will not be (de)serialized. Defaults to None.

• clean (bool) – If true, all files in the store will be removed recursively

• **storage_options – Keyword arguments passed to filesystem initializer.

complete_key(partial_key: pathlib.Path, **open_kwargs) → List[str]

Returns a list of possible key continuations given a partial key.

get(key: pathlib.Path, mode: str = 'rb', **open_kwargs) → Any

Retrieves an item with the given key.

key_exists(key: pathlib.Path, **open_kwargs) → bool

Checks if a key exists.

set(value: Any, key: Optional[pathlib.Path] = None, mode: str = 'wb', **open_kwargs) → None

Persists an item with the given key.

class squirrel.artifact_manager.filesystem.FileSystemArtifactManager(url: str, serializer: Optional[squirrel.serialization.SquirrelSerializer] = None, collection: str = 'default', **fs_kwargs)

Bases: squirrel.artifact_manager.base.ArtifactManager

Helper class that provides a standard way to create an ABC using inheritance.

Artifactmanager backed by fsspec filesystems.

The manager logs artifacts according to the following file structure: url/collection/artifact/version/serializer/<content>

url: root directory of the artifact store collection: the name of the collection defaults to ‘default’ artifact: (human-readable) name of the artifact version: version number starting at 1 which is automatically incremented serializer: the name of the serializer used to store the artifact (e.g. file, messagepack)

collection_to_catalog(collection: Optional[str] = None) → squirrel.catalog.Catalog

Provide catalog of all artifacts and their versions contained within specific collection

download_artifact(artifact: str, collection: Optional[str] = None, version: Optional[str] = None, to: Optional[pathlib.Path] = None) → Union[pathlib.Path, squirrel.artifact_manager.base.TmpArtifact]

Download artifact to local path.

exists_in_collection(artifact: str, collection: Optional[str] = None) → bool

Check if artifact exists in specified collection.

abstract get_artifact(artifact: str, collection: Optional[str] = None, version: Optional[str] = None) → Any

Retrieve specific artifact value.

get_artifact_source(artifact: str, collection: Optional[str] = None, version: Optional[str] = None, catalog: Optional[squirrel.catalog.Catalog] = None) → squirrel.catalog.catalog.CatalogSource

Catalog entry for a specific artifact

list_collection_names() → Iterable

List all collections managed by this ArtifactManager.

abstract log_artifact(obj: Any, name: str, collection: Optional[str] = None) → squirrel.catalog.catalog.Source

Log an arbitrary python object using store serialisation.

log_files(artifact_name: str, local_path: pathlib.Path, collection: Optional[str] = None, artifact_path: Optional[pathlib.Path] = None) → squirrel.catalog.catalog.CatalogSource

Upload local file or folder to artifact store without serialisation