squirrel.serialization.msgpack

Module Contents

Classes

MessagepackSerializer

Helper class that provides a standard way to create an ABC using
class squirrel.serialization.msgpack.MessagepackSerializer

Bases: squirrel.serialization.serializer.SquirrelSerializer

Helper class that provides a standard way to create an ABC using inheritance.

static deserialize(obj: bytes) → Any

Returns the object deserialized with msgpack.

static deserialize_shard_from_file(fp: str, fs: Optional[fsspec.spec.AbstractFileSystem] = None, mode: str = 'rb', unpacker_kwargs: Optional[Dict] = None, **open_kwargs) → Iterable[Any]

Reads a shard from file and returns an iterable over its samples.

Parameters

  • fp (str) – Path to the file to write.

  • fs (AbstractFileSystem, optional) – Filesystem to use for opening the file. If not provided, fsspec will pick a filesystem suitable for fp. Defaults to None.

  • mode (str) – IO mode to use. Passed to fs.open(). Defaults to “rb”.

  • unpacker_kwargs (Dict, optional) – Kwargs to be passed to msgpack.Unpacker(). If use_list not given, it will be set to False.

  • **open_kwargs – Other keyword arguments passed to fs.open(). open_kwargs will always have compression=”gzip” set.

Yields

(Any) Values of the samples of the shard.

static serialize(obj: Any) → bytes

Returns the object serialized with msgpack.

static serialize_shard_to_file(shard: squirrel.constants.ShardType, fp: str, fs: Optional[fsspec.spec.AbstractFileSystem] = None, mode: str = 'wb', **open_kwargs) → None

Writes a shard to a file by only writing and serializing the values of its samples.

Parameters

  • shard (ShardType) – Shard to serialize and write to the file.

  • fp (str) – Path to the file to write.

  • fs (AbstractFileSystem, optional) – Filesystem to use for opening the file. If not provided, fsspec will pick a filesystem suitable for fp. Defaults to None.

  • mode (str) – IO mode to use. Passed to fs.open(). Defaults to “wb”.

  • **open_kwargs – Other keyword arguments passed to fs.open(). open_kwargs will always have compression=”gzip” set.