bloscpack 0.11.0

Basics

Basic compression:

$ blpk compress data.dat

Or:

$ blpk c data.dat

… will compress the file data.dat to data.dat.blp

Basic decompression:

$ blpk decompress data.dat.blp data.dcmp

Or:

$ blpk d data.dat.blp data.dcmp

… will decompress the file data.dat.blp to the file data.dcmp. If you leave out the [<out_file>] argument, Bloscpack will complain that the file data.dat exists already and refuse to overwrite it:

$ blpk decompress data.dat.blp
blpk: error: output file 'data.dat' exists!

If you know what you are doing, you can use the global option [-f | --force] to override the overwrite checks:

$ blpk --force decompress data.dat.blp

Incidentally this works for compression too:

$ blpk compress data.dat
blpk: error: output file 'data.dat.blp' exists!
$ blpk --force compress data.dat

Lastly, if you want a different filename:

$ blpk compress data.dat custom.filename.blp

… will compress the file data.dat to custom.filename.blp

Settings

By default, the number of threads that Blosc uses during compression and decompression is determined by the number of cores detected on your system. You can change this using the [-n | --nthreads] option:

$ blpk --nthreads 1 compress data.dat

Compression with Blosc is controlled with the following options:

• [-t | --typesize] Typesize used by Blosc (default: 8): $ blpk compress --typesize 8 data.dat

• [-l | --level] Compression level (default: 7): $ blpk compress --level 3 data.dat

• [-s | --no-shuffle] Deactivate shuffle: $ blpk compress --no-shuffle data.dat

• [-c | --codec] Use alternative codec: $ blpk compress --codec lz4 data.dat

In addition, there are the following options that control the Bloscpack file:

• [-z | --chunk-size] Desired approximate size of the chunks, where you can use human readable strings like 8M or 128K or max to use the maximum chunk size of apprx. 2GB (default: 1MB): $ blpk compress --chunk-size 128K data.dat or $ blpk c -z max data.dat

• [-k | --checksum <checksum>] Chose which checksum to use. The following values are permissible: None, adler32, crc32, md5, sha1, sha224, sha256, sha384, sha512, (default: adler32). As described in the header format, each compressed chunk can be stored with a checksum, which aids corruption detection on decompression: $ blpk compress --checksum crc32 data.dat

• [-o | --no-offsets] By default, offsets to the individual chunks are stored. These are included to allow for partial decompression in the future. This option disables that feature. Also, a certain number of offsets (default: 10 * ‘nchunks’) are preallocated to allow for appending data to the file: $ blpk compress --no-offsets data.dat

Info Subcommand

If you just need some info on how the file was compressed [i | info]:

$ blpk info data.dat.blp
blpk: BloscpackHeader:
blpk:     format_version: 3
blpk:     offsets: True
blpk:     metadata: False
blpk:     checksum: 'adler32'
blpk:     typesize: 8
blpk:     chunk_size: 512.0M (536870912B)
blpk:     last_chunk: 501.88M (526258176B)
blpk:     nchunks: 3
blpk:     max_app_chunks: 30
blpk: 'offsets':
blpk: [296,78074317,140782616,...]

Adding Metdata

Using the [-m | --metadata] option you can include JSON from a file:

$ cat meta.json
{"dtype": "float64", "shape": [200000000], "container": "numpy"}
$ blpk compress --chunk-size=512M --metadata meta.json data.dat
$ blpk info data.dat.blp
blpk: BloscpackHeader:
blpk:     format_version: 3
blpk:     offsets: True
blpk:     metadata: True
blpk:     checksum: 'adler32'
blpk:     typesize: 8
blpk:     chunk_size: 512.0M (536870912B)
blpk:     last_chunk: 501.88M (526258176B)
blpk:     nchunks: 3
blpk:     max_app_chunks: 30
blpk: 'offsets':
blpk: [922,78074943,140783242,...]
blpk: 'metadata':
blpk: {   u'container': u'numpy', u'dtype': u'float64', u'shape': [200000000]}
blpk: MetadataHeader:
blpk:     magic_format: 'JSON'
blpk:     meta_options: '00000000'
blpk:     meta_checksum: 'adler32'
blpk:     meta_codec: 'zlib'
blpk:     meta_level: 6
blpk:     meta_size: 59.0B (59B)
blpk:     max_meta_size: 590.0B (590B)
blpk:     meta_comp_size: 58.0B (58B)
blpk:     user_codec: ''

It will be printed when decompressing:

$ blpk decompress data.dat.blp
blpk: Metadata is:
blpk: '{u'dtype': u'float64', u'shape': [200000000], u'container': u'numpy'}'

Appending

You can also append data to an existing bloscpack compressed file:

$ blpk append data.dat.blp data.dat

However there are certain limitations on the amount of data can be appended. For example, if there is an offsets section, there must be enough room to store the offsets for the appended chunks. If no offsets exists, you may append as much data as possible given the limitations governed by the maximum number of chunks and the chunk-size. Additionally, there are limitations on the compression options. For example, one cannot change the checksum used. It is however possible to change the compression level, the typesize and the shuffle option for the appended chunks.

Also note that appending is still considered experimental as of v0.5.0.

Verbose and Debug mode

Lastly there are two mutually exclusive options to control how much output is produced.

The first causes basic info to be printed, [-v | --verbose]:

$ blpk --verbose compress --chunk-size 0.5G data.dat
blpk: using 4 threads
blpk: getting ready for compression
blpk: input file is: 'data.dat'
blpk: output file is: 'data.dat.blp'
blpk: input file size: 1.49G (1600000000B)
blpk: nchunks: 3
blpk: chunk_size: 512.0M (536870912B)
blpk: last_chunk_size: 501.88M (526258176B)
blpk: output file size: 198.39M (208028617B)
blpk: compression ratio: 7.691250
blpk: done

… and [-d | --debug] prints a detailed account of what is going on:

$ blpk --debug compress --chunk-size 0.5G data.dat
blpk: command line argument parsing complete
blpk: command line arguments are:
blpk:     force: False
blpk:     verbose: False
blpk:     offsets: True
blpk:     checksum: adler32
blpk:     subcommand: compress
blpk:     out_file: None
blpk:     metadata: None
blpk:     cname: blosclz
blpk:     in_file: data.dat
blpk:     chunk_size: 536870912
blpk:     debug: True
blpk:     shuffle: True
blpk:     typesize: 8
blpk:     clevel: 7
blpk:     nthreads: 4
blpk: using 4 threads
blpk: getting ready for compression
blpk: input file is: 'data.dat'
blpk: output file is: 'data.dat.blp'
blpk: input file size: 1.49G (1600000000B)
blpk: nchunks: 3
blpk: chunk_size: 512.0M (536870912B)
blpk: last_chunk_size: 501.88M (526258176B)
blpk: BloscArgs:
blpk:     typesize: 8
blpk:     clevel: 7
blpk:     shuffle: True
blpk:     cname: 'blosclz'
blpk: BloscpackArgs:
blpk:     offsets: True
blpk:     checksum: 'adler32'
blpk:     max_app_chunks: <function <lambda> at 0x1182de8>
blpk: metadata_args will be silently ignored
blpk: max_app_chunks is a callable
blpk: max_app_chunks was set to: 30
blpk: BloscpackHeader:
blpk:     format_version: 3
blpk:     offsets: True
blpk:     metadata: False
blpk:     checksum: 'adler32'
blpk:     typesize: 8
blpk:     chunk_size: 512.0M (536870912B)
blpk:     last_chunk: 501.88M (526258176B)
blpk:     nchunks: 3
blpk:     max_app_chunks: 30
blpk: raw_bloscpack_header: 'blpk\x03\x01\x01\x08\x00\x00\x00 \x00\x10^\x1f\x03\x00\x00\x00\x00\x00\x00\x00\x1e\x00\x00\x00\x00\x00\x00\x00'
blpk: Handle chunk '0'
blpk: checksum (adler32): '\x1f\xed\x1e\xf4'
blpk: chunk handled, in: 512.0M (536870912B) out: 74.46M (78074017B)
blpk: Handle chunk '1'
blpk: checksum (adler32): ')\x1e\x08\x88'
blpk: chunk handled, in: 512.0M (536870912B) out: 59.8M (62708295B)
blpk: Handle chunk '2' (last)
blpk: checksum (adler32): '\xe8\x18\xa4\xac'
blpk: chunk handled, in: 501.88M (526258176B) out: 64.13M (67245997B)
blpk: Writing '3' offsets: '[296, 78074317, 140782616]'
blpk: Raw offsets: '(\x01\x00\x00\x00\x00\x00\x00\xcdQ\xa7\x04\x00\x00\x00\x00\x18,d\x08\x00\x00\x00\x00'
blpk: output file size: 198.39M (208028617B)
blpk: compression ratio: 7.691250
blpk: done

Numpy

Numpy arrays can be serialized as Bloscpack files, here is a very brief example:

>>> a = np.linspace(0, 1, 3e8)
>>> print a.size, a.dtype
300000000 float64
>>> bp.pack_ndarray_file(a, 'a.blp')
>>> b = bp.unpack_ndarray_file('a.blp')
>>> (a == b).all()
True

Looking at the generated file, we can see the Numpy metadata being saved:

$ lh a.blp
-rw------- 1 esc esc 266M Aug 13 23:21 a.blp

$ blpk info a.blp
blpk: BloscpackHeader:
blpk:     format_version: 3
blpk:     offsets: True
blpk:     metadata: True
blpk:     checksum: 'adler32'
blpk:     typesize: 8
blpk:     chunk_size: 1.0M (1048576B)
blpk:     last_chunk: 838.0K (858112B)
blpk:     nchunks: 2289
blpk:     max_app_chunks: 22890
blpk: 'offsets':
blpk: [202170,408064,554912,690452,819679,...]
blpk: 'metadata':
blpk: {   u'container': u'numpy',
blpk:     u'dtype': u'<f8',
blpk:     u'order': u'C',
blpk:     u'shape': [300000000]}
blpk: MetadataHeader:
blpk:     magic_format: 'JSON'
blpk:     meta_options: '00000000'
blpk:     meta_checksum: 'adler32'
blpk:     meta_codec: 'zlib'
blpk:     meta_level: 6
blpk:     meta_size: 67.0B (67B)
blpk:     max_meta_size: 670.0B (670B)
blpk:     meta_comp_size: 62.0B (62B)
blpk:     user_codec: ''

Alternatively, we can also use a string as storage:

>>> a = np.linspace(0, 1, 3e8)
>>> c = pack_ndarray_str(a)
>>> b = unpack_ndarray_str(c)
>>> (a == b).all()
True

Or use alternate compressors:

>>> a = np.linspace(0, 1, 3e8)
>>> c = pack_ndarray_str(a, blosc_args=BloscArgs(cname='lz4'))
>>> b = unpack_ndarray_str(c)
>>> (a == b).all()
True

If you are interested in the performance of Bloscpack compared to other serialization formats for Numpy arrays, please look at the benchmarks presented in the Bloscpack paper from the EuroScipy 2013 conference proceedings.

Installing Dependencies

Testing requires some additional libraries, which you can install from PyPi with:

$ pip install -r test_requirements.txt
[...]

Basic Tests

Basic tests, runs quickly:

$ nosetests
[...]

Heavier Tests

Extended tests using a larger file, may take some time, but will be nice to memory:

$ nosetests test/test_file_io.py:pack_unpack_hard
[...]

Extended tests using a huge file. This one take forever and needs loads (5G-6G) of memory and loads of disk-space (10G). Use -s to print progress:

$ nosetests -s test/test_file_io.py:pack_unpack_extreme
[...]

Note that, some compression/decompression tests create temporary files (on UNIXoid systems this is under /tmp/blpk*) which are deleted upon completion of the respective test, both successful and unsuccessful, or when the test is aborted with e.g. ctrl-c (using atexit magic).

Under rare circumstances, for example when aborting the deletion which is triggered on abort you may be left with large files polluting your temporary space. Depending on your partitioning scheme etc.. doing this repeatedly, may lead to you running out of space on the file-system.

Command Line Interface Tests

The command line interface is tested with cram:

$ cram --verbose test_cmdline/*.cram
[...]

Coverage

To determine coverage you can pool together the coverage from the cram tests and the unit tests:

$ COVERAGE=1 cram --verbose test_cmdline/*.cram
[...]
$nosetests --with-coverage --cover-package=bloscpack
[...]

Test Runner

To run the command line interface tests and the unit tests and analyse coverage, use the convenience test.sh runner:

$ ./test.sh
[...]

Description of the header

The following 32 bit header is used for Bloscpack as of version 0.3.0. The design goals of the header format are to contain as much information as possible to achieve interesting things in the future and to be as general as possible such that the persistence layer of Blaze/BLZ can be implemented without modification of the header format.

The following ASCII representation shows the layout of the header:

|-0-|-1-|-2-|-3-|-4-|-5-|-6-|-7-|-8-|-9-|-A-|-B-|-C-|-D-|-E-|-F-|
| b   l   p   k | ^ | ^ | ^ | ^ |   chunk-size  |  last-chunk   |
                  |   |   |   |
      version ----+   |   |   |
      options --------+   |   |
     checksum ------------+   |
     typesize ----------------+

|-0-|-1-|-2-|-3-|-4-|-5-|-6-|-7-|-8-|-9-|-A-|-B-|-C-|-D-|-E-|-F-|
|            nchunks            |        max-app-chunks         |

The first 4 bytes are the magic string blpk. Then there are 4 bytes which hold information about the activated features in this file. This is followed by 4 bytes for the chunk-size, another 4 bytes for the last-chunk-size, 8 bytes for the number of chunks, nchunks and lastly 8 bytes for the total number of chunks that can be appended to this file, max-app-chunks.

Effectively, storing the number of chunks as a signed 8 byte integer, limits the number of chunks to 2**63-1 = 9223372036854775807, but this should not be relevant in practice, since, even with the moderate default value of 1MB for chunk-size, we can still store files as large as 8ZB (!) Given that in 2012 the maximum size of a single file in the Zettabye File System (zfs) is 16EB, Bloscpack should be safe for a few more years.

Description of the header entries

All entries are little-endian.

version:

(uint8) format version of the Bloscpack header, to ensure exceptions in case of forward incompatibilities.

options:

(bitfield) A bitfield which allows for setting certain options in this file.

bit 0 (0x01):

If the offsets to the chunks are present in this file.

bit 1 (0x02):

If metadata is present in this file.

checksum:

(uint8) The checksum used. The following checksums, available in the python standard library should be supported. The checksum is always computed on the compressed data and placed after the chunk.

0:

no checksum

1:

zlib.adler32

2:

zlib.crc32

3:

hashlib.md5

4:

hashlib.sha1

5:

hashlib.sha224

6:

hashlib.sha256

7:

hashlib.sha384

8:

hashlib.sha512

typesize:

(uint8) The typesize of the data in the chunks. Currently, assume that the typesize is uniform. The space allocated is the same as in the Blosc header.

chunk-size:

(int32) Denotes the chunk-size. Since the maximum buffer size of Blosc is 2GB having a signed 32 bit int is enough (2GB = 2**31 bytes). The special value of -1 denotes that the chunk-size is unknown or possibly non-uniform.

last-chunk:

(int32) Denotes the size of the last chunk. As with the chunk-size an int32 is enough. Again, -1 denotes that this value is unknown.

nchunks:

(int64) The total number of chunks used in the file. Given a chunk-size of one byte, the total number of chunks is 2**63. This amounts to a maximum file-size of 8EB (8EB = 2*63 bytes) which should be enough for the next couple of years. Again, -1 denotes that the number of is unknown.

max-app-chunks:

(int64) The maximum number of chunks that can be appended to this file, excluding nchunks. This is only useful if there is an offsets section and if nchunks is known (not -1), if either of these conditions do not apply this should be 0.

The overall file-size can be computed as chunk-size * (nchunks - 1) + last-chunk-size. In a streaming scenario -1 can be used as a placeholder. For example if the total number of chunks, or the size of the last chunk is not known at the time the header is created.

The following constraints exist on the header entries:

• last-chunk must be less than or equal to chunk-size.

• nchunks + max_app_chunks must be less than or equal to the maximum value of an int64.

Description of the metadata section

This section goes after the header. It consists of a metadata-section header followed by a serialized and potentially compressed data section, followed by preallocated space to resize the data section, possibly followed by a checksum.

The layout of the section is thus:

|-metadata-header-|-data-|-prealloc-|-checksum-|

The header has the following layout:

|-0-|-1-|-2-|-3-|-4-|-5-|-6-|-7-|-8-|-9-|-A-|-B-|-C-|-D-|-E-|-F-|
|         magic-format          | ^ | ^ | ^ | ^ |   meta-size   |
                                  |   |   |   |
              meta-options -------+   |   |   |
              meta-checksum ----------+   |   |
              meta-codec -----------------+   |
              meta-level ---------------------+

|-0-|-1-|-2-|-3-|-4-|-5-|-6-|-7-|-8-|-9-|-A-|-B-|-C-|-D-|-E-|-F-|
| max-meta-size |meta-comp-size |            user-codec         |

magic-format:

(8 byte ASCII string) The data will usually be some kind of binary serialized string data, for example JSON, BSON, YAML or Protocol-Buffers. The format identifier is to be placed in this field.

meta-options:

(bitfield) A bitfield which allows for setting certain options in this metadata section. Currently unused

meta-checksum:

The checksum used for the metadata. The same checksums as for the data are available.

meta-codec:

(unit8) The codec used for compressing the metadata. As of Bloscpack version 0.3.0 the following codecs are supported.

0:

no codec

1:

zlib (DEFLATE)

meta-level:

(unit8) The compression level used for the codec. If codec is 0 i.e. the metadata is not compressed, this must be 0 too.

meta-size:

(uint32) The size of the uncompressed metadata.

max-meta-size:

(uint32) The total allocated space for the data section.

meta-comp-size:

(uint32) If the metadata is compressed, this gives the total space the metadata occupies. If the data is not compressed this is the same as meta-size. In a sense this is the true amount of space in the metadata section that is used.

user-codec:

Space reserved for usage of additional codecs. E.g. 4 byte magic string for codec identification and 4 bytes for encoding of codec parameters.

The total space left for enlarging the metadata section is simply: max-meta-size - meta-comp-size.

JSON Example of serialized metadata:

'{"dtype": "float64", "shape": [1024], "others": []}'

If compression is requested, but not beneficial, because the compressed size would be larger than the uncompressed size, compression of the metadata is automatically deactivated.

As of Bloscpack version 0.3.0 only the JSON serializer is supported and used the string JSON followed by four whitespace bytes as identifier. Since JSON and any other of the suggested serializers has limitations, only a subset of Python structures can be stored, so probably some additional object handling must be done prior to serialize certain kinds of metadata.

Description of the offsets entries

Following the metadata section, comes a variable length section of chunk offsets. Offsets of the chunks into the file are to be used for accelerated seeking. The offsets (if activated) follow the header. Each offset is a 64 bit signed little-endian integer (int64). A value of -1 denotes an unknown offset. Initially, all offsets should be initialized to -1 and filled in after writing all chunks. Thus, If the compression of the file fails prematurely or is aborted, all offsets should have the value -1. Also, any unused offset entries preallocated to allow the file to grow should be set to -1. Each offset denotes the exact position of the chunk in the file such that seeking to the offset, will position the file pointer such that, reading the next 16 bytes gives the Blosc header, which is at the start of the desired chunk.

Description of the chunk format

As mentioned previously, each chunk is just a Blosc compressed string including header. The Blosc header (as of v1.0.0) is 16 bytes as follows:

|-0-|-1-|-2-|-3-|-4-|-5-|-6-|-7-|-8-|-9-|-A-|-B-|-C-|-D-|-E-|-F-|
  ^   ^   ^   ^ |     nbytes    |   blocksize   |    ctbytes    |
  |   |   |   |
  |   |   |   +--typesize
  |   |   +------flags
  |   +----------versionlz
  +--------------version

The first four are simply bytes, the last three are are each unsigned ints (uint32) each occupying 4 bytes. The header is always little-endian. ctbytes is the length of the buffer including header and nbytes is the length of the data when uncompressed. A more detailed description of the Blosc header can be found in the README_HEADER.rst of the Blosc repository

Overhead

Depending on which configuration for the file is used a constant, or linear overhead may be added to the file. The Bloscpack header adds 32 bytes in any case. If the data is non-compressible, Blosc will add 16 bytes of header to each chunk. The metadata section obviously adds a constant overhead, and if used, both the checksum and the offsets will add overhead to the file. The offsets add 8 bytes per chunk and the checksum adds a fixed constant value which depends on the checksum to each chunk. For example, 32 bytes for the adler32 checksum.

Documentation

• Refactor monolithic readme into Sphinx and publish

• Write the docstrings for the Args classes

• Cleanup and double check the docstrings for the public API classes

• document library usage

• Announcement RST

Command Line

• quiet verbosity level

• Expose the ability to set ‘max_app_chunks’ from the command line

• Allow to save metadata to a file during decompression

• subcommand e or estimate to estimate the size of the uncompressed data.

• subcommand v or verify to verify the integrity of the data

• add –raw-input and –raw-output switches to allow stuff like: cat file | blpk –raw-input –raw-output compress > file.blp

• Establish and document proper exit codes

• Document the metadata saved during Numpy serialization

Profiling and Optimization

• Use the faster version of struct where you have a single string

• Memory profiler, might be able to reduce memory used by reusing the buffer during compression and decompression

• Benchmark different codecs

• Use line profiler to check code

• Select different defaults for Numpy arrays, no offsets? no pre-alloc?

Library Features

• possibly provide a BloscPackFile abstraction, like GzipFile

• Allow to not-prealloc additional space for metadata

• Refactor certain collections of functions that operate on data into objects

  • Offsets (maybe)

• partial decompression?

• since we now have potentially small chunks, the progressbar becomes relevant again

• configuration file to store commonly used options on a given machine

• print the compression time, either as verbose or debug

• Investigate if we can use a StringIO object that returns memoryviews on read.

• Implement a memoryview Compressed/PlainSource

• Use a bytearray to read chunks from a file. Then re-use that bytearray during every read to avoid allocating deallocating strings the whole time.

• The keyword arguments to many functions are global dicts, this is a bad idea, Make the immutable with a forzendict.

• Check that the checksum is really being checked for all PlainSinks

• Bunch of NetworkSource/Sinks

• HTTPSource/Sink

Miscellaneous

• Announce on scipy/numpy lists, comp.compression, freshmeat, ohloh …

Packaging and Infrastructure

• Debian packages (for python-blosc and bloscpack)

• Conda recipes (for python-blosc and bloscpack)

• Use tox for testing multiple python versions

• Build on travis and drone.io using pre-compiled