Simple dataset to dataloader library for pytorch
Project description
This is a simple library for creating readable dataset pipelines and reusing best practices for issues such as imbalanced datasets. There are just two components to keep track of: Dataset and Datastream.
Dataset is a simple mapping between an index and an example. It provides pipelining of functions in a readable syntax originally adapted from tensorflow 2’s tf.data.Dataset.
Datastream combines a Dataset and a sampler into a stream of examples. It provides a simple solution to oversampling / stratification, weighted sampling, and finally converting to a torch.utils.data.DataLoader.
Install
poetry add pytorch-datastreamOr, for the old-timers:
pip install pytorch-datastreamUsage
The list below is meant to showcase functions that are useful in most standard and non-standard cases. It is not meant to be an exhaustive list. See the documentation for a more extensive list on API and usage.
Dataset.from_subscriptable
Dataset.from_dataframe
Dataset
.map
.subset
.split
.cache
.with_columns
Datastream.merge
Datastream.zip
Datastream
.map
.data_loader
.zip_index
.update_weights_
.update_example_weight_
.weight
.state_dict
.load_state_dictMerge / stratify / oversample datastreams
The fruit datastreams given below repeatedly yields the string of its fruit type.
>>> datastream = Datastream.merge([
... (apple_datastream, 2),
... (pear_datastream, 1),
... (banana_datastream, 1),
... ])
>>> next(iter(datastream.data_loader(batch_size=8)))
['apple', 'apple', 'pear', 'banana', 'apple', 'apple', 'pear', 'banana']Zip independently sampled datastreams
The fruit datastreams given below repeatedly yields the string of its fruit type.
>>> datastream = Datastream.zip([
... apple_datastream,
... Datastream.merge([pear_datastream, banana_datastream]),
... ])
>>> next(iter(datastream.data_loader(batch_size=4)))
[('apple', 'pear'), ('apple', 'banana'), ('apple', 'pear'), ('apple', 'banana')]More usage examples
See the documentation for more usage examples.
Install from source
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pytorch_datastream-0.4.10.tar.gz
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 2943cf82091090d1d459cf4e7f3ac7a242f038efb95709dc0b4e08f5c7896bfd |
|
| MD5 | 373abf64ad2c0e53b7f31fd8531a49d5 |
|
| BLAKE2b-256 | edd37f5481bbe604973c4995b2734019a4ffa3220a7f520f8b62bbd18c98fb8b |
Hashes for pytorch_datastream-0.4.10-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 89afca7f52fe24351caf69bcba44c2e5c2a1bfba05e0421da27d80006ef4283a |
|
| MD5 | e67bf36fbe4f82e3e9046fdf97fa6429 |
|
| BLAKE2b-256 | dfd55067dd96e96aea49737f7b7cd411645d76bb2327fb495ffb89c82de431fe |
