squirrel.serialization.jsonl
¶
Module Contents¶
Classes¶
Helper class that provides a standard way to create an ABC using |
|
Json decoder for numpy types. |
|
Json encoder for numpy types. |
-
class
squirrel.serialization.jsonl.
JsonSerializer
(deser_hook: Optional[Callable] = None)¶ Bases:
squirrel.serialization.serializer.SquirrelSerializer
Helper class that provides a standard way to create an ABC using inheritance.
Initializes JsonSerializer.
- Parameters
deser_hook (Callable) – Callable that is passed as object_hook to
JsonDecoder
during json deserialization. Defaults to None.
-
static
deserialize_shard_from_file
(fp: str, fs: Optional[fsspec.spec.AbstractFileSystem] = None, mode: str = 'rb', **open_kwargs) → Iterable[Any]¶ Reads a shard from file and returns an iterable over its samples.
- Parameters
fp (str) – Path to the file to write.
fs (AbstractFileSystem, optional) – Filesystem to use for opening the file. If not provided, fsspec will pick a filesystem suitable for fp. Defaults to None.
mode (str) – IO mode to use. Passed to
fs.open()
. Defaults to “rb”.**open_kwargs – Other keyword arguments passed to
fs.open()
. open_kwargs will always have compression=”gzip” set.
- Yields
(Any) Values of the samples of the shard.
-
static
serialize_shard_to_file
(shard: squirrel.constants.ShardType, fp: str, fs: Optional[fsspec.spec.AbstractFileSystem] = None, mode: str = 'wb', **open_kwargs) → None¶ Writes a shard to a file by only writing and serializing the values of its samples.
- Parameters
shard (ShardType) – Shard to serialize and write to the file.
fp (str) – Path to the file to write.
fs (AbstractFileSystem, optional) – Filesystem to use for opening the file. If not provided, fsspec will pick a filesystem suitable for fp. Defaults to None.
mode (str) – IO mode to use. Passed to
fs.open()
. Defaults to “wb”.**open_kwargs – Other keyword arguments passed to
fs.open()
. open_kwargs will always have compression=”gzip” set.
-
class
squirrel.serialization.jsonl.
SquirrelJsonDecoder
(*args, **kwargs)¶ Bases:
json.JSONDecoder
Json decoder for numpy types.
Initialize SquirrelJsonDecoder.
-
object_hook
(self, dct: Dict) → Any¶ Decode custom types.
-
-
class
squirrel.serialization.jsonl.
SquirrelJsonEncoder
(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)¶ Bases:
json.JSONEncoder
Json encoder for numpy types.
Constructor for JSONEncoder, with sensible defaults.
If skipkeys is false, then it is a TypeError to attempt encoding of keys that are not str, int, float or None. If skipkeys is True, such items are simply skipped.
If ensure_ascii is true, the output is guaranteed to be str objects with all incoming non-ASCII characters escaped. If ensure_ascii is false, the output can contain non-ASCII characters.
If check_circular is true, then lists, dicts, and custom encoded objects will be checked for circular references during encoding to prevent an infinite recursion (which would cause an OverflowError). Otherwise, no such check takes place.
If allow_nan is true, then NaN, Infinity, and -Infinity will be encoded as such. This behavior is not JSON specification compliant, but is consistent with most JavaScript based encoders and decoders. Otherwise, it will be a ValueError to encode such floats.
If sort_keys is true, then the output of dictionaries will be sorted by key; this is useful for regression tests to ensure that JSON serializations can be compared on a day-to-day basis.
If indent is a non-negative integer, then JSON array elements and object members will be pretty-printed with that indent level. An indent level of 0 will only insert newlines. None is the most compact representation.
If specified, separators should be an (item_separator, key_separator) tuple. The default is (‘, ‘, ‘: ‘) if indent is
None
and (‘,’, ‘: ‘) otherwise. To get the most compact JSON representation, you should specify (‘,’, ‘:’) to eliminate whitespace.If specified, default is a function that gets called for objects that can’t otherwise be serialized. It should return a JSON encodable version of the object or raise a
TypeError
.-
default
(self, obj: Any) → Any¶ The default function to encode numpy types.
-