squirrel.serialization
¶
Submodules¶
Package Contents¶
Classes¶
Helper class that provides a standard way to create an ABC using |
|
Helper class that provides a standard way to create an ABC using |
|
Helper class that provides a standard way to create an ABC using |
-
class
squirrel.serialization.
JsonSerializer
(deser_hook: Optional[Callable] = None)¶ Bases:
squirrel.serialization.serializer.SquirrelSerializer
Helper class that provides a standard way to create an ABC using inheritance.
Initializes JsonSerializer.
- Parameters
deser_hook (Callable) – Callable that is passed as object_hook to
JsonDecoder
during json deserialization. Defaults to None.
-
static
deserialize_shard_from_file
(fp: str, fs: Optional[fsspec.spec.AbstractFileSystem] = None, mode: str = 'rb', **open_kwargs) → Iterable[Any]¶ Reads a shard from file and returns an iterable over its samples.
- Parameters
fp (str) – Path to the file to write.
fs (AbstractFileSystem, optional) – Filesystem to use for opening the file. If not provided, fsspec will pick a filesystem suitable for fp. Defaults to None.
mode (str) – IO mode to use. Passed to
fs.open()
. Defaults to “rb”.**open_kwargs – Other keyword arguments passed to
fs.open()
. open_kwargs will always have compression=”gzip” set.
- Yields
(Any) Values of the samples of the shard.
-
static
serialize_shard_to_file
(shard: squirrel.constants.ShardType, fp: str, fs: Optional[fsspec.spec.AbstractFileSystem] = None, mode: str = 'wb', **open_kwargs) → None¶ Writes a shard to a file by only writing and serializing the values of its samples.
- Parameters
shard (ShardType) – Shard to serialize and write to the file.
fp (str) – Path to the file to write.
fs (AbstractFileSystem, optional) – Filesystem to use for opening the file. If not provided, fsspec will pick a filesystem suitable for fp. Defaults to None.
mode (str) – IO mode to use. Passed to
fs.open()
. Defaults to “wb”.**open_kwargs – Other keyword arguments passed to
fs.open()
. open_kwargs will always have compression=”gzip” set.
-
class
squirrel.serialization.
MessagepackSerializer
¶ Bases:
squirrel.serialization.serializer.SquirrelSerializer
Helper class that provides a standard way to create an ABC using inheritance.
-
static
deserialize_shard_from_file
(fp: str, fs: Optional[fsspec.spec.AbstractFileSystem] = None, mode: str = 'rb', unpacker_kwargs: Optional[Dict] = None, **open_kwargs) → Iterable[Any]¶ Reads a shard from file and returns an iterable over its samples.
- Parameters
fp (str) – Path to the file to write.
fs (AbstractFileSystem, optional) – Filesystem to use for opening the file. If not provided, fsspec will pick a filesystem suitable for fp. Defaults to None.
mode (str) – IO mode to use. Passed to
fs.open()
. Defaults to “rb”.unpacker_kwargs (Dict, optional) – Kwargs to be passed to msgpack.Unpacker(). If use_list not given, it will be set to False.
**open_kwargs – Other keyword arguments passed to
fs.open()
. open_kwargs will always have compression=”gzip” set.
- Yields
(Any) Values of the samples of the shard.
-
static
serialize_shard_to_file
(shard: squirrel.constants.ShardType, fp: str, fs: Optional[fsspec.spec.AbstractFileSystem] = None, mode: str = 'wb', **open_kwargs) → None¶ Writes a shard to a file by only writing and serializing the values of its samples.
- Parameters
shard (ShardType) – Shard to serialize and write to the file.
fp (str) – Path to the file to write.
fs (AbstractFileSystem, optional) – Filesystem to use for opening the file. If not provided, fsspec will pick a filesystem suitable for fp. Defaults to None.
mode (str) – IO mode to use. Passed to
fs.open()
. Defaults to “wb”.**open_kwargs – Other keyword arguments passed to
fs.open()
. open_kwargs will always have compression=”gzip” set.
-
static
-
class
squirrel.serialization.
SquirrelSerializer
¶ Bases:
abc.ABC
Helper class that provides a standard way to create an ABC using inheritance.
-
abstract
deserialize
(self, obj: Any) → Any¶ Returns the deserialized object.
-
deserialize_shard_from_file
(self, fp: str, **kwargs) → Iterable[squirrel.constants.SampleType]¶ Reads a shard from file and returns an iterable over the values of its samples.
-
abstract
serialize
(self, obj: Any) → Any¶ Returns the serialized object.
-
serialize_shard_to_file
(self, obj: squirrel.constants.ShardType, fp: str, **kwargs) → None¶ Serializes a list of samples and writes it to a file.
-
abstract