Skip to main content

Fork of Python 3 pickle module.

Project description

zodbpickle README

This package presents a uniform pickling interface for ZODB:

  • Under Python2, this package forks both Python 2.7’s pickle and cPickle modules, adding support for the protocol 3 opcodes. It also provides a new subclass of bytes, zodbpickle.binary, which Python2 applications can use to pickle binary values such that they will be unpickled as bytes under Py3k.

  • Under Py3k, this package forks the pickle moudule (and the supporting C extension) from both Python 3.2 and Python 3.3. The fork add support for the noload operations used by ZODB.

Loading/Storing Python 2 Strings

In all their wisdom, the Python developers have decided that Python 2 str instances should be loaded as Python 3 str objects (i.e. unicode strings). Patches were proposed in Python issue 6784 but were never applied. This code base contains those patches.

Example 1: Loading Python 2 pickles on Python 3

$ python2
>>> import pickle
>>> pickle.dumps('\xff', protocol=0)
"S'\\xff'\np0\n."
>>> pickle.dumps('\xff', protocol=1)
'U\x01\xffq\x00.'
>>> pickle.dumps('\xff', protocol=2)
'\x80\x02U\x01\xffq\x00.'

$ python3
>>> from zodbpickle import pickle
>>> pickle.loads(b"S'\\xff'\np0\n.", encoding='bytes')
b'\xff'
>>> pickle.loads(b'U\x01\xffq\x00.', encoding='bytes')
b'\xff'
>>> pickle.loads(b'\x80\x02U\x01\xffq\x00.', encoding='bytes')
b'\xff'

Example 2: Loading Python 3 pickles on Python 2

$ python3
>>> from zodbpickle import pickle
>>> pickle.dumps(b"\xff", protocol=0)
b'c_codecs\nencode\np0\n(V\xff\np1\nVlatin1\np2\ntp3\nRp4\n.'
>>> pickle.dumps(b"\xff", protocol=1)
b'c_codecs\nencode\nq\x00(X\x02\x00\x00\x00\xc3\xbfq\x01X\x06\x00\x00\x00latin1q\x02tq\x03Rq\x04.'
>>> pickle.dumps(b"\xff", protocol=2)
b'\x80\x02c_codecs\nencode\nq\x00X\x02\x00\x00\x00\xc3\xbfq\x01X\x06\x00\x00\x00latin1q\x02\x86q\x03Rq\x04.'

$ python2
>>> import pickle
>>> pickle.loads('c_codecs\nencode\np0\n(V\xff\np1\nVlatin1\np2\ntp3\nRp4\n.')
'\xff'
>>> pickle.loads('c_codecs\nencode\nq\x00(X\x02\x00\x00\x00\xc3\xbfq\x01X\x06\x00\x00\x00latin1q\x02tq\x03Rq\x04.')
'\xff'
>>> pickle.loads('\x80\x02c_codecs\nencode\nq\x00X\x02\x00\x00\x00\xc3\xbfq\x01X\x06\x00\x00\x00latin1q\x02\x86q\x03Rq\x04.')
'\xff'

Example 3: everything breaks down

$ python2
>>> class Foo(object):
...     def __init__(self):
...         self.x = 'hello'
...
>>> import pickle
>>> pickle.dumps(Foo(), protocol=0)
"ccopy_reg\n_reconstructor\np0\n(c__main__\nFoo\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'x'\np6\nS'hello'\np7\nsb."
>>> pickle.dumps(Foo(), protocol=1)
'ccopy_reg\n_reconstructor\nq\x00(c__main__\nFoo\nq\x01c__builtin__\nobject\nq\x02Ntq\x03Rq\x04}q\x05U\x01xq\x06U\x05helloq\x07sb.'
>>> pickle.dumps(Foo(), protocol=2)
'\x80\x02c__main__\nFoo\nq\x00)\x81q\x01}q\x02U\x01xq\x03U\x05helloq\x04sb.'

$ python3
>>> from zodbpickle import pickle
>>> class Foo(object): pass
...
>>> foo = pickle.loads("ccopy_reg\n_reconstructor\np0\n(c__main__\nFoo\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'x'\np6\nS'hello'\np7\nsb.", encoding='bytes')
>>> foo.x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Foo' object has no attribute 'x'

wait what?

>>> foo.__dict__
{b'x': b'hello'}

oooh. So we use encoding='ASCII' (the default) and errors='bytes' and hope it works:

>>> foo = pickle.loads("ccopy_reg\n_reconstructor\np0\n(c__main__\nFoo\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'x'\np6\nS'hello'\np7\nsb.", errors='bytes')
>>> foo.x
'hello'

falling back to bytes if necessary

>>> pickle.loads(b'\x80\x02U\x01\xffq\x00.', errors='bytes')
b'\xff'

Support for noload()

The ZODB uses cPickle’s noload() method to retrieve all persistent references from a pickle without loading any objects. This feature was removed from Python 3’s pickle. Unfortuantely, this unnecessarily fills the pickle cache.

This module provides a noload() method again.

zodbpickle Changelog

0.5.0 (2013-06-14)

  • Removed support for the bytes_as_strings arguments to pickling APIs: the pickles created when that argument was true might not be unpickled without passing encoding='bytes', which ZODB couldn’t reliably enforce. On Py3k, ZODB will be using protocol=3 pickles anyway.

0.4.4 (2013-06-07)

  • Add protocol 3 opcodes to the C version of the noload() dispatcher.

0.4.3 (2013-06-07)

  • Packaging error: remove spurious -ASIDE file from sdist.

0.4.2 (2013-06-07)

  • Fix NameError in pure-Python version of Unpickler.noload_appends.

  • Fix NameError in pure-Python version of Unpickler.noload_setitems.

0.4.1 (2013-04-29)

  • Fix typo in Python2 version of zodbpickle.pickle module.

0.4 (2013-04-28)

  • Support the common pickle module interface for Python 2.6, 2.7, 3.2, and 3.3.

  • Split the Python implementations / tests into Python2- and Py3k-specific variants.

  • Added a fork of the Python 2.7 _pickle.c, for use under Python2. The fork adds support for the Py3k protocol 3 opcodes.

  • Added a custom binary type for use in Python2 apps. Derived from bytes, the binary type allows Python2 apps to pickle binary data using opcodes which will cause it to be unpickled as bytes on Py3k. Under Py3k, the binary type is just an alias for bytes.

0.3 (2013-03-18)

  • Added noload code to Python 3.2 version of Unpickler. As with the Python 3.3 version, this code remains untested.

  • Added bytes_as_strings option to the Python 3.2 version of Pickler, dump, and dumps.

0.2 (2013-03-05)

  • Added bytes_as_strings option to Pickler, dump, and dumps.

  • Incomplete support for Python 3.2:

    • Move _pickle.c -> _pickle_33.c.

    • Clone Python 3.2.3’s _pickle.c -> _pickle_32.c and apply the same patch.

    • Choose between them at build time based on sys.version_info.

    • Disable some tests of 3.3-only features.

    • Missing: implementation of noload() in _pickle_32.c.

    • Missing: implementation of bytes_as_strings=True in _pickle_32.c.

0.1.0 (2013-02-27)

  • Initial release of Python 3.3’s pickle with the patches of Python issue 6784 applied.

  • Added support for errors="bytes".

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

zodbpickle-0.5.0.tar.gz (215.5 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page