test_torch_distibuted_loading
¶
Module Contents¶
Functions¶
|
Helper function to create test data. |
|
Quick sanity test that the data can be loaded and a fixed number of samples taken. |
|
Fixture for this modules test data |
|
Test that we can load the data across multiple PyTorch dataloaders workers in a single node setup |
|
Test we can load the data correctly in a multi-node and multi-worker setting. |
Attributes¶
-
test_torch_distibuted_loading.
MAX_SAMPLES_PER_SHARD
= 100¶
-
test_torch_distibuted_loading.
MIN_SAMPLES_PER_SHARD
= 50¶
-
test_torch_distibuted_loading.
N_SHARDS
= 20¶
-
test_torch_distibuted_loading.
create_data
(test_folder: str) → int¶ Helper function to create test data.
-
test_torch_distibuted_loading.
test_JSONSource
(test_data: Generator) → None¶ Quick sanity test that the data can be loaded and a fixed number of samples taken.
-
test_torch_distibuted_loading.
test_data
() → Iterator[Tuple[str, int]]¶ Fixture for this modules test data