squirrel.driver.jsonl

Module Contents

Classes

JsonlDriver

A StoreDriver that by default uses SquirrelStore with jsonl serialization.
class squirrel.driver.jsonl.JsonlDriver(url: str, deser_hook: Optional[Callable] = None, **kwargs)

Bases: squirrel.driver.store_driver.StoreDriver

A StoreDriver that by default uses SquirrelStore with jsonl serialization.

Initializes JsonlDriver with default store and serializer.

Parameters

  • url (str) – Path to the root directory. If this path does not exist, it will be created.

  • deser_hook (Callable) – Callable that is passed as object_hook to JsonDecoder during json deserialization. Defaults to None.

  • **kwargs – Keyword arguments passed to the super class initializer.

name = jsonl
get_iter(self, get_kwargs: Optional[Dict] = None, **kwargs) → squirrel.iterstream.Composable

Returns an iterable of samples as specified by fetcher_func.

Parameters

  • get_kwargs (Dict) – Keyword arguments that will be passed as get_kwargs to MapDriver.get_iter(). get_kwargs will always have compression=”gzip”. Defaults to None.

  • **kwargs – Other keyword arguments that will be passed to MapDriver.get_iter().

Returns

(Composable) Iterable over the items in the store.