A Python class that supports Data Science projects.
Project description
resumableds
A Python class that supports Data Science projects.
resumableds supports you in writing data science scripts including save/resume functionality.
Data can be saved and resumed avoiding unnessary retrievals of raw data from data storages.
The data directory structure is inspired by cookiecutter-data-science (https://drivendata.github.io/cookiecutter-data-science/).
The class also supports the statement 'Analysis is a DAG' (https://drivendata.github.io/cookiecutter-data-science/#analysis-is-a-dag).
resumableds is written in pure Python and it is intended to be used within Jupyter notebooks. It however can also be useful in Python scripts or script pipelines.
Example
proj1 = RdsProject('project1') # create object from class (creates the dir if it doesn't exist yet)
proj1.raw.df1 = pd.DataFrame() # create dataframe as attribute of proj1.raw (RdsFs 'raw')
proj1.defs.variable1 = 'foo' # create simple objects as attribute of proj1.defs (RdsFs 'defs')
proj1.save() # saved attributes of all RfdFs in proj1 to disk
This will result in the following directory structure (plus some overhead of internals):
- <output_dir>/defs/var_variable1.pkl
- <output_dir>/raw/df1.pkl
- <output_dir>/raw/df1.csv
Note, pandas dataframes are always dumped as pickle for further processing and as csv for easy exploration. The csv files are never read back anymore.
Later on or in another python session, you can do this:
proj2 = RdsProject('project1') # vars and data are read back to their original names
proj2.defs.variable1 == 'foo' # ==> True
isinstance(proj2.raw.df1, pd.DataFrame) # ==> True
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for resumableds-0.4.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d79db45a64dc3897cf273b5c587d6350d2bc4cc33455b6fe0e1e41043dd0c6ae |
|
MD5 | 568803648dc579b2c5a6b58616fecc68 |
|
BLAKE2b-256 | 688b1c86014f88bb3d424e0b0abc7acbaf37eda012ea04e86e4518a04806addd |