Tafra: innards of a dataframe
Project description
The tafra began life as a thought experiment: how could we reduce the idea of a dataframe (as expressed in libraries like pandas or languages like R) to its useful essence, while carving away the cruft? The original proof of concept stopped at “group by”.
This library expands on the proof of concept to produce a practically useful tafra, which we hope you may find to be a helpful lightweight substitute for certain uses of pandas.
A tafra is, more-or-less, a set of named columns or dimensions. Each of these is a typed numpy array of consistent length, representing the values for each column by rows.
The library provides lightweight syntax for manipulating rows and columns, support for managing data types, iterators for rows and sub-frames, pandas-like “transform” support and conversion from pandas Dataframes, and SQL-style “group by” and join operations.
Tafra |
|
Aggregations |
Union, GroupBy, Transform, IterateBy, InnerJoin, LeftJoin, CrossJoin |
Aggregation Helpers |
union, union_inplace, group_by, transform, iterate_by, inner_join, left_join, cross_join |
Constructors |
|
Destructors |
|
Properties |
|
Iter Methods |
|
Dict-like Methods |
keys, values, items, get, update, update_inplace, update_dtypes, update_dtypes_inplace |
Other Helper Methods |
rename, rename_inplace, coalesce, coalesce_inplace, _coalesce_dtypes, delete, delete_inplace |
Printer Methods |
Getting Started
Install the library with pip:
pip install tafraA short example
>>> from tafra import Tafra
>>> t = Tafra({
... 'x': np.array([1, 2, 3, 4]),
... 'y': np.array(['one', 'two', 'one', 'two'], dtype='object'),
... })
>>> t.pformat()
Tafra(data = {
'x': array([1, 2, 3, 4]),
'y': array(['one', 'two', 'one', 'two'])},
dtypes = {
'x': 'int', 'y': 'object'},
rows = 4)
>>> print('List:', '\n', t.to_list())
List:
[array([1, 2, 3, 4]), array(['one', 'two', 'one', 'two'], dtype=object)]
>>> print('Records:', '\n', tuple(t.to_records()))
Record:
((1, 'one'), (2, 'two'), (3, 'one'), (4, 'two'))
>>> gb = t.group_by(
... ['y'], {'x': sum}
... )
>>> print('Group By:', '\n', gb.pformat())
Group By:
Tafra(data = {
'x': array([4, 6]), 'y': array(['one', 'two'])},
dtypes = {
'x': 'int', 'y': 'object'},
rows = 2)Flexibility
Have some code that works with pandas, or just a way of doing things that you prefer? tafra is flexible:
>>> df = pd.DataFrame(np.c_[
... np.array([1, 2, 3, 4]),
... np.array(['one', 'two', 'one', 'two'])
... ], columns=['x', 'y'])
>>> t = Tafra.from_dataframe(df)And going back is just as simple:
>>> df = pd.DataFrame(t.data)Timings
In this case, lightweight also means performant. Beyond any additional features added to the library, tafra should provide the necessary base for organizing data structures for numerical processing. One of the most important aspects is fast access to the data itself. By minizing abstraction to access the underlying numpy arrays, tafra provides over an order of magnitude increase in performance.
Import note If you assign directly to the Tafra.data or Tafra._data attributes, you must call Tafra._coalesce_dtypes afterwards in order to ensure the typing is consistent.
Construct a Tafra and a DataFrame:
>>> t = Tafra({
... 'x': np.array([1, 2, 3, 4]),
... 'y': np.array(['one', 'two', 'one', 'two'], dtype='object'),
... })
>>> df = pd.DataFrame(t.data)Read Operations
Direct access:
>>> %timemit x = t._data['x']
55.3 ns ± 5.64 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)Indirect with some penalty to support Tafra slicing and numpy’s advanced indexing:
>>> %timemit x = t['x']
219 ns ± 71.6 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)pandas timing:
>>> %timemit x = df['x']
1.55 µs ± 105 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)As fast as pandas gets:
>>> where_col = list(df.columns).index('x')
>>> %timeit x = df.values[:, where_col]
48 µs ± 7.77 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)Assignment Operations
Direct access:
>>> x = np.arange(4)
>>> %timeit tf._data['x'] = x
65 ns ± 5.55 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)Indidrect:
>>> %timeit tf['x'] = x
7.39 µs ± 950 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)pandas timing:
>>> %timeit df['x'] = x
47.8 µs ± 3.53 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)Version History
1.0.1
Add iter functions
Add map functions
Various constructor improvements
1.0.0
Initial Release
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
