zappend

zappend is a tool written in Python that is used for robustly creating and updating Zarr datacubes from smaller dataset slices. It is built on top of the awesome Python packages xarray and zarr.

Motivation

The objective of zappend is to address recurring memory issues when generating large geospatial datacubes using the Zarr format by subsequently concatenating data slices along an append dimension, e.g., time (the default) for geospatial satellite observations. Each append step is atomic, that is, the append operation is a transaction that can be rolled back, in case the append operation fails. This ensures integrity of the target data cube.

Features

The zappend tool provides the following features:

• Locking: While the target dataset is being modified, a file lock is created, effectively preventing concurrent dataset modifications.
• Transaction-based dataset appends: On failure during an append step, the transaction is rolled back, so that the target dataset remains valid and preserves its integrity.
• Filesystem transparency: The target dataset may be generated and updated in any writable filesystems supported by the fsspec package. The same holds for the slice datasets to be appended.
• Dataset polling: The tool can be configured to wait for slice datasets to become available.
• CLI and Python API: The tool can be used in a shell using the zappend command or from Python. When used from Python using the zappend() function, slice datasets can be passed as local file paths, URIs, as datasets of type xarray.Dataset, or as custom zappend.api.SliceSource objects.

More about zappend can be found in its documentation.