image source. antipickle keeps your heterogeneous data fresh

antipickle

when you want to use pickle, but you shouldn't

Why? Because pickle isn't the right way to persist or share data, and we all know that.

When it comes to practice, it takes time and effort to substitute pickle.
'Hmm, I can use json here' — I thought on many occasions, and usually was wrong.

Something small but annoying was in the way:
datetime that can't be stored or np.array that serializers don't know how to deal with. Or even bytes! And many smaller things.

At this point I either had to give up and pickle it
OR allocate time on figuring out 'how do I make this right'.

antipickle solves this for me.

antipickle is a restricted format for safe, persistent, and platform-independent storage.

Also, it is very convenient:

import antipickle
antipickle.dump(data, 'data.antipickle')
antipickle.dump(data, 's3://mybucket/data.antipickle')     # stores in s3 
antipickle.dump(data, 's3://mybucket/data.antipickle.gz')  # will additionally gzip


loaded_date = antipickle.load('s3://mybucket/data.antipickle.gz') # or local file

To download/upload from s3 you need an additional dependency: pip install s3fs.

Batteries included:

Here is a simple example of what antipickle can save/load:

data = {
    'constants': [3.1415, 2.718, True, False, 42],
    'with nones': [1, None, 0],
    b'bytes': b"can be stored too!",
    'nested lists and tuples': [[1, [2]], (1, 2, None), {'nested': 'dict'}],
    ('tuple', 'as', 'key'): {'is_ok': True},
    'numpy nd': np.zeros([3, 4], dtype='uint32'),
}
antipickle.dump(data, 'data.antipickle')

More formally, antipickle supports python pieces commonly used for computations:

• bytes, str, int, float, complex, bool, and None
• list, tuple, set (all of them are stored as different entities)
• dict (including integer keys and tuple keys)
• numpy arrays (native .npy format used; dtype=O not supported)
• pandas series and dataframe (using parquet serialization via pyarrow)
• polars series and dataframe (using parquet)
• Any tree-formed structure of the above (no loops allowed)

Configurable support: dataclasses and pydantic classes.

For reference, other non-pythonic formats (json and its binary relatives) have problems with native types (not making difference between list and tuple) or encodings (not storing bytes) or collections (not allowing integers, bytes and tuples in dict keys).
Antipickle is python-centric and has it solved.

Installation

pip install antipickle

What is it for

Let's set the expectation bar. antipickle is

• not fast, but isn't slow either
• not super-compact, but quite ok
• restricted: it wasn't designed to serialize just anything, it focuses on common python types and cases for data folks

At the same time, antipickle is

• safe
• persistent
• very convenient
• modular and easy to extend

and thus suitable for data sharing and data preservation.

When to (not) use pickle

pickle is designed for interprocess communication or as a temporary storage. pickle has a good tradeoff of space- and time- efficiency and can serialize almost anything, including graphs with cycles.

Name pickle suggests you could use it for long-term preservation of data, but that's not true: pickle's serialization is tied to an internal object representation, which is not guaranteed to be preserved in the next release (or even on a different OS). Developers of some packages (notably scikit-learn) provide some guarantees about being able to parse models that were saved with previous 1-2 minor package releases, but that's an exception not a rule.

Second, pickle is insecure. And unreadable. And pickles can be large. During unpickling they can do anything python can, i.e. anything at all. So python docs say it clear: Only unpickle data you trust!.

That said, pickle is extremely convenient and simple to use, and works as a short-term solution for many cases, so we all (python data folks) kinda doing that wrong pickling thing from time to time, because of convenience. And because very few of us are ready to spend time on figuring out proper serialization.

All comments above apply to pickle-like libs like joblib, dill, cloudpickle.

License

antipickle is distributed under the terms of the MIT license.

Other

antipickle builds upon msgpack-python (the only dependency).

antipickle supports all maintained python versions (Python 3.7+)