`squirrel.artifact_manager.base`¶

Module Contents¶

Classes¶

`ArtifactManager`	Helper class that provides a standard way to create an ABC using
`DirectoryLogger`	Class to be used as a context for logging an entire directory as an artifact.
`TmpArtifact`	Class to be used as a context for temporarily downloading an artifact, interacting with it and then deleting it.

Attributes¶

logger

squirrel.artifact_manager.base.logger¶

class squirrel.artifact_manager.base.ArtifactManager(collection: str = 'default')¶

Bases: abc.ABC

Helper class that provides a standard way to create an ABC using inheritance.

Artifact manager interface for various backends.

property active_collection → str ¶

Collections act as folders of artifacts.

It is ultimately up to the user how to structure their artifact store and the collections therein. All operations accessing artifacts allow explicit specification of the collection to use. Therefore, users could use collections to separate different artifact types or different experiments / runs. The manager maintains an ‘active’ collection which it logs to by default. This can be set at the run start when the manager is initialized and then remain unchanged for the duration of the run.

To avoid incompatibility between different backends, collections cannot be nested (e.g. as subfolders on a filesystem) as in particular the WandB backend has no real notion of nested folder structures.

abstract collection_to_catalog(collection: Optional[str] = None) → squirrel.catalog.Catalog ¶: Catalog of all artifacts within a specific collection.

abstract download_artifact(artifact: str, collection: Optional[str] = None, version: Optional[str] = None, to: Optional[pathlib.Path] = None) → Union[pathlib.Path, TmpArtifact]¶

Download artifact contents (from current collection) to specific location and return a source listing them.

If no target location is specified, a context manager for a temporary directory is created and the path to it is returned. Retrieve latest version unless specified.

download_collection(collection: Optional[str] = None, to: pathlib.Path = './') → squirrel.catalog.Catalog ¶: Download all artifacts in collection to local directory.

exists(artifact: str) → bool ¶: Check if artifact exists in specified collection.

abstract exists_in_collection(artifact: str, collection: Optional[str] = None) → bool ¶: Check if artifact exists in specified collection.

abstract get_artifact(artifact: str, collection: Optional[str] = None, version: Optional[str] = None) → Any¶: Retrieve specific artifact value.

abstract list_collection_names() → Iterable¶: Return list of all collections in the artifact store

abstract log_artifact(obj: Any, name: str, collection: Optional[str] = None) → squirrel.catalog.catalog.CatalogSource ¶

Log an arbitrary python object

The serialisation method used is backend dependent. When using a simple FileStore backend any SquirrelSerializer can be chosen. For WandB objects serialisation is handled by WandB itself.

abstract log_files(artifact_name: str, local_path: pathlib.Path, collection: Optional[str] = None, artifact_path: Optional[pathlib.Path] = None) → squirrel.catalog.catalog.CatalogSource ¶

Upload a file or folder into (current) collection, increment version automatically

Parameters

artifact_name – Name of artifact to log
local_path – Local path to the file or folder to log
collection – Name of collection to log to, defaults to the active collection
artifact_path – path under which to log the files within the artifact, defaults to “./”

log_folder(artifact: str, collection: Optional[str] = None) → DirectoryLogger ¶: Create a context manager for logging a directory of files as a single artifact.

store_to_catalog() → squirrel.catalog.Catalog ¶: Provide Catalog of all artifacts stored in backend.

class squirrel.artifact_manager.base.DirectoryLogger(artifact_manager: ArtifactManager, artifact: str, collection: Optional[str] = None)¶

Class to be used as a context for logging an entire directory as an artifact. When entering the scope it creates a local dir with a valid afid and returns the filepath to it. You can then write files to that dir and after exiting the scope the dir gets logged through the artifact manager.

Initializes the DirectoryLogger.

Parameters

artifact_manager – An artifact manager instance to use for logging the final artifact.
artifact – The name of the artifact to log.
collection – The name of the collection to log to, defaults to the active collection of the artifact manager.

__enter__() → pathlib.Path ¶

Called when entering the context. Creates folder under /tmp with the artifacts id if it doesn’t exist yet.

Returns: Absolute path to artifact folder as str

__exit__(exctype: Optional[Type[BaseException]] = None, excinst: Optional[BaseException] = None, exctb: Optional[types.TracebackType] = None) → None ¶: Called when exiting the context. Logs the artifact folder if it’s not empty.

class squirrel.artifact_manager.base.TmpArtifact(artifact_manager: ArtifactManager, collection: str, artifact: str, version: str)¶

Class to be used as a context for temporarily downloading an artifact, interacting with it and then deleting it. When entering the scope it downloads the artifact to a local dir and returns the filepath to it.

Initializes the TmpArtifact.

Parameters

artifact_manager – An artifact manager instance to use for downloading the artifact.
collection – The name of the collection to download from.
artifact – The name of the artifact to download.
version – The version of the artifact to download.

__enter__() → pathlib.Path ¶

Called when entering the context. Downloads the artifact to a temporary dir and returns the filepath to it.

Returns: Absolute path to artifact folder

__exit__(exctype: Optional[Type[BaseException]] = None, excinst: Optional[BaseException] = None, exctb: Optional[types.TracebackType] = None) → None ¶: Called when exiting the context. Deletes the artifact folder.

squirrel.artifact_manager.base¶

Module Contents¶

Classes¶

Attributes¶

`squirrel.artifact_manager.base`¶