SeqTools 1.1.0

Example

>>> def f1(x):
... return x + 1
...
>>> def f2(x): # slow and memory heavy transformation
... time.sleep(.01)
... return [x for _ in range(500)]
...
>>> def f3(x):
... return sum(x) / len(x)
...
>>> data = list(range(1000))

Without delayed evaluation, defining the pipeline and reading values looks like so:

>>> tmp1 = [f1(x) for x in data]
>>> tmp2 = [f2(x) for x in tmp1] # takes 10 seconds and a lot of memory
>>> res = [f3(x) for x in tmp2]
>>> print(res[2])
3.0
>>> print(max(tmp2[2])) # requires to store 499 500 useless values along
3

With seqtools:

>>> tmp1 = seqtools.smap(f1, data)
>>> tmp2 = seqtools.smap(f2, tmp1)
>>> res = seqtools.smap(f3, tmp2) # no computations so far
>>> print(res[2]) # takes 0.01 seconds
3.0
>>> print(max(tmp2[2])) # easy access to intermediate results
3

Batteries included!

The library comes with a set of functions to manipulate sequences:

concatenation	batching	reindexing
prefetching	interleaving

and others (suggestions are also welcome).

Installation

pip install seqtools

Documentation

The documentation is hosted at https://seqtools-doc.readthedocs.io.

Contributing and Support

Use the issue tracker to request features, propose improvements or report issues. For questions regarding usage, please send an email.

Related libraries

Joblib, proposes low-level functions with many optimization settings to optimize pipelined transformations. This library notably provides advanced caching mechanisms which are not the primary concern of SeqTool. SeqTool uses a simpler container-oriented interface with multiple utility functions in order to assist fast prototyping. On-demand evaluation is its default behaviour and applies at all layers of a transformation pipeline. In particular, parallel evaluation can be inserted in the middle of the transformation pipeline and won’t block the execution to wait for the computation of all elements from the dataset.

SeqTools is conceived to connect nicely to the data loading pipeline of Machine Learning libraries such as PyTorch’s torch.utils.data and torchvision.transforms or Tensorflow’s tf.data. The interface of these libraries focuses on iterators to access transformed elements, contrary to SeqTools which also provides arbitrary reads via indexing.