squirrel.serialization

Submodules

• squirrel.serialization.jsonl
• squirrel.serialization.msgpack
• squirrel.serialization.serializer

Package Contents

Classes

JsonSerializer	Helper class that provides a standard way to create an ABC using
MessagepackSerializer	Helper class that provides a standard way to create an ABC using
SquirrelSerializer	Helper class that provides a standard way to create an ABC using

class squirrel.serialization.JsonSerializer(deser_hook: Optional[Callable] = None)

Bases: squirrel.serialization.serializer.SquirrelSerializer

Helper class that provides a standard way to create an ABC using inheritance.

Initializes JsonSerializer.

Parameters

deser_hook (Callable) – Callable that is passed as object_hook to JsonDecoder during json deserialization. Defaults to None.

deserialize(obj: bytes) → Any

Returns the object deserialized with json.

static deserialize_shard_from_file(fp: str, fs: Optional[fsspec.spec.AbstractFileSystem] = None, mode: str = 'rb', **open_kwargs) → Iterable[Any]

Reads a shard from file and returns an iterable over its samples.

Parameters

• fp (str) – Path to the file to write.

• fs (AbstractFileSystem, optional) – Filesystem to use for opening the file. If not provided, fsspec will pick a filesystem suitable for fp. Defaults to None.

• mode (str) – IO mode to use. Passed to fs.open(). Defaults to “rb”.

• **open_kwargs – Other keyword arguments passed to fs.open(). open_kwargs will always have compression=”gzip” set.

Yields

(Any) Values of the samples of the shard.

static serialize(obj: Any) → bytes

Returns the object serialized with json.

static serialize_shard_to_file(shard: squirrel.constants.ShardType, fp: str, fs: Optional[fsspec.spec.AbstractFileSystem] = None, mode: str = 'wb', **open_kwargs) → None

Writes a shard to a file by only writing and serializing the values of its samples.

Parameters

• shard (ShardType) – Shard to serialize and write to the file.

• fp (str) – Path to the file to write.

• fs (AbstractFileSystem, optional) – Filesystem to use for opening the file. If not provided, fsspec will pick a filesystem suitable for fp. Defaults to None.

• mode (str) – IO mode to use. Passed to fs.open(). Defaults to “wb”.

• **open_kwargs – Other keyword arguments passed to fs.open(). open_kwargs will always have compression=”gzip” set.

class squirrel.serialization.MessagepackSerializer

Bases: squirrel.serialization.serializer.SquirrelSerializer

Helper class that provides a standard way to create an ABC using inheritance.

static deserialize(obj: bytes) → Any

Returns the object deserialized with msgpack.

static deserialize_shard_from_file(fp: str, fs: Optional[fsspec.spec.AbstractFileSystem] = None, mode: str = 'rb', unpacker_kwargs: Optional[Dict] = None, **open_kwargs) → Iterable[Any]

Reads a shard from file and returns an iterable over its samples.

Parameters

• fp (str) – Path to the file to write.

• fs (AbstractFileSystem, optional) – Filesystem to use for opening the file. If not provided, fsspec will pick a filesystem suitable for fp. Defaults to None.

• mode (str) – IO mode to use. Passed to fs.open(). Defaults to “rb”.

• unpacker_kwargs (Dict, optional) – Kwargs to be passed to msgpack.Unpacker(). If use_list not given, it will be set to False.

• **open_kwargs – Other keyword arguments passed to fs.open(). open_kwargs will always have compression=”gzip” set.

Yields

(Any) Values of the samples of the shard.

static serialize(obj: Any) → bytes

Returns the object serialized with msgpack.

static serialize_shard_to_file(shard: squirrel.constants.ShardType, fp: str, fs: Optional[fsspec.spec.AbstractFileSystem] = None, mode: str = 'wb', **open_kwargs) → None

Writes a shard to a file by only writing and serializing the values of its samples.

Parameters

• shard (ShardType) – Shard to serialize and write to the file.

• fp (str) – Path to the file to write.

• fs (AbstractFileSystem, optional) – Filesystem to use for opening the file. If not provided, fsspec will pick a filesystem suitable for fp. Defaults to None.

• mode (str) – IO mode to use. Passed to fs.open(). Defaults to “wb”.

• **open_kwargs – Other keyword arguments passed to fs.open(). open_kwargs will always have compression=”gzip” set.

class squirrel.serialization.SquirrelSerializer

Bases: abc.ABC

Helper class that provides a standard way to create an ABC using inheritance.

abstract deserialize(obj: Any) → Any

Returns the deserialized object.

deserialize_shard_from_file(fp: str, **kwargs) → Iterable[squirrel.constants.SampleType]

Reads a shard from file and returns an iterable over the values of its samples.

abstract serialize(obj: Any) → Any

Returns the serialized object.

serialize_shard_to_file(obj: squirrel.constants.ShardType, fp: str, **kwargs) → None

Serializes a list of samples and writes it to a file.