squirrel.artifact_manager.filesystem
¶
Module Contents¶
Classes¶
A FilesystemStore serving as the backend for the FileSystemArtifactManager. |
|
Helper class that provides a standard way to create an ABC using |
Attributes¶
-
squirrel.artifact_manager.filesystem.
Serializers
¶
-
class
squirrel.artifact_manager.filesystem.
ArtifactFileStore
(url: str, serializer: Optional[squirrel.serialization.SquirrelSerializer] = None, clean: bool = False, **storage_options)¶ Bases:
squirrel.store.FilesystemStore
A FilesystemStore serving as the backend for the FileSystemArtifactManager.
- The get and set methods are altered to allow for storing serialized data as well as raw files.
If the final path component is a serializer name, the data is stored as a serialized file. If the final path component is “files”, the data is stored as a raw file.
Initializes FilesystemStore.
- Parameters
url (str) – Path to the root directory. If this path does not exist, it will be created.
serializer (SquirrelSerializer, optional) – Serializer that is used to serialize data before persisting (see
set()
) and to deserialize data after reading (seeget()
). If not specified, data will not be (de)serialized. Defaults to None.clean (bool) – If true, all files in the store will be removed recursively
**storage_options – Keyword arguments passed to filesystem initializer.
-
complete_key
(partial_key: pathlib.Path, **open_kwargs) → List[str]¶ Returns a list of possible key continuations given a partial key.
-
get
(key: pathlib.Path, mode: str = 'rb', **open_kwargs) → Any¶ Retrieves an item with the given key.
-
key_exists
(key: pathlib.Path, **open_kwargs) → bool¶ Checks if a key exists.
-
set
(value: Any, key: Optional[pathlib.Path] = None, mode: str = 'wb', **open_kwargs) → None¶ Persists an item with the given key.
-
class
squirrel.artifact_manager.filesystem.
FileSystemArtifactManager
(url: str, serializer: Optional[squirrel.serialization.SquirrelSerializer] = None, collection: str = 'default', **fs_kwargs)¶ Bases:
squirrel.artifact_manager.base.ArtifactManager
Helper class that provides a standard way to create an ABC using inheritance.
Artifactmanager backed by fsspec filesystems.
The manager logs artifacts according to the following file structure: url/collection/artifact/version/serializer/<content>
url: root directory of the artifact store collection: the name of the collection defaults to ‘default’ artifact: (human-readable) name of the artifact version: version number starting at 1 which is automatically incremented serializer: the name of the serializer used to store the artifact (e.g. file, messagepack)
-
collection_to_catalog
(collection: Optional[str] = None) → squirrel.catalog.Catalog¶ Provide catalog of all artifacts and their versions contained within specific collection
-
download_artifact
(artifact: str, collection: Optional[str] = None, version: Optional[str] = None, to: Optional[pathlib.Path] = None) → Union[pathlib.Path, squirrel.artifact_manager.base.TmpArtifact]¶ Download artifact to local path.
-
exists_in_collection
(artifact: str, collection: Optional[str] = None) → bool¶ Check if artifact exists in specified collection.
-
abstract
get_artifact
(artifact: str, collection: Optional[str] = None, version: Optional[str] = None) → Any¶ Retrieve specific artifact value.
-
get_artifact_source
(artifact: str, collection: Optional[str] = None, version: Optional[str] = None, catalog: Optional[squirrel.catalog.Catalog] = None) → squirrel.catalog.catalog.CatalogSource¶ Catalog entry for a specific artifact
-
list_collection_names
() → Iterable¶ List all collections managed by this ArtifactManager.
-
abstract
log_artifact
(obj: Any, name: str, collection: Optional[str] = None) → squirrel.catalog.catalog.Source¶ Log an arbitrary python object using store serialisation.
-
log_files
(artifact_name: str, local_path: pathlib.Path, collection: Optional[str] = None, artifact_path: Optional[pathlib.Path] = None) → squirrel.catalog.catalog.CatalogSource¶ Upload local file or folder to artifact store without serialisation
-