Streaming operations with pandas.
Project description
pandas_streaming: streaming API over pandas
pandas_streaming aims at processing big files with pandas, too big to hold in memory, too small to be parallelized with a significant gain. The module replicates a subset of pandas API and implements other functionalities for machine learning.
from pandas_streaming.df import StreamingDataFrame sdf = StreamingDataFrame.read_csv("filename", sep="\t", encoding="utf-8") for df in sdf: # process this chunk of data # df is a dataframe print(df)
The module can also stream an existing dataframe.
import pandas df = pandas.DataFrame([dict(cf=0, cint=0, cstr="0"), dict(cf=1, cint=1, cstr="1"), dict(cf=3, cint=3, cstr="3")]) from pandas_streaming.df import StreamingDataFrame sdf = StreamingDataFrame.read_df(df) for df in sdf: # process this chunk of data # df is a dataframe print(df)
It contains other helpers to split datasets into train and test with some weird constraints.
Links:
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pandas_streaming-0.3.218.tar.gz
(33.3 kB
view hashes)
Built Distribution
Close
Hashes for pandas_streaming-0.3.218-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a4e7a8d923598f7d3fe5e999c8f543c061c4e77a5eb48b9c66b2a169c03ad19b |
|
MD5 | 0e9e5e5fe2f947beab9d23d405a837b2 |
|
BLAKE2b-256 | a9d7b58ebf4810b4324cea00b72a423a65bf6c2bf083b1ecd7ad3fc06d174373 |