A modern C++ header only cdf library
Project description
Python packages
Linux x86_64 | Windows x86_64 | MacOs x86_64 | MacOs ARM64 |
---|---|---|---|
Unit Tests
Linux x86_64 | Windows x86_64 | MacOs x86_64 |
---|---|---|
CDFpp (CDF++)
A NASA's CDF modern C++ library. This is not a C++ wrapper but a full C++ implementation. Why? CDF files are still used for space physics missions but few implementations are available. The main one is NASA's C implementation available here but it lacks multi-threads support (global shared state), has an old C interface and has a license which isn't compatible with most Linux distributions policy. There are also Java and Python implementations which are not usable in C++.
List of features and roadmap:
- CDF reading
- read uncompressed file headers
- read uncompressed attributes
- read uncompressed variables
- read variable attributes
- loads cdf files from memory (std::vector or char*)
- handles both row and column major files
- read variables with nested VXRs
- read compressed files (GZip, RLE)
- read compressed variables (GZip, RLE)
- read UTF-8 encoded files
- read ISO 8859-1(Latin-1) encoded files (converts to UTF-8 on the fly)
- variables values lazy loading
- decode DEC's floating point encoding (Itanium, ALPHA and VAX)
- pad values
- CDF writing
- write uncompressed headers
- write uncompressed attributes
- write uncompressed variables
- write compressed variables
- write compressed files
- pad values
- General features
- uses libdeflate for faster GZip decompression
- highly optimized CDF reads (up to ~4GB/s read speed from disk)
- handle leap seconds
- Python wrappers
- Documentation
- Examples (see below)
- Benchmarks
If you want to understand how it works, how to use the code or what works, you may have to read tests.
Installing
From PyPi
python3 -m pip install --user pycdfpp
From sources
meson build
cd build
ninja
sudo ninja install
Or if youl want to build a Python wheel:
python -m build .
# resulting wheel will be located into dist folder
Basic usage
Python
Reading CDF files
Basic example from a local file:
import pycdfpp
cdf = pycdfpp.load("some_cdf.cdf")
cdf_var_data = cdf["var_name"].values #builds a numpy view or a list of strings
attribute_name_first_value = cdf.attributes['attribute_name'][0]
Note that you can also load in memory files:
import pycdfpp
import requests
import matplotlib.pyplot as plt
tha_l2_fgm = pycdfpp.load(requests.get("https://spdf.gsfc.nasa.gov/pub/data/themis/tha/l2/fgm/2016/tha_l2_fgm_20160101_v01.cdf").content)
plt.plot(tha_l2_fgm["tha_fgl_gsm"])
plt.show()
Buffer protocol support:
import pycdfpp
import requests
import xarray as xr
import matplotlib.pyplot as plt
tha_l2_fgm = pycdfpp.load(requests.get("https://spdf.gsfc.nasa.gov/pub/data/themis/tha/l2/fgm/2016/tha_l2_fgm_20160101_v01.cdf").content)
xr.DataArray(tha_l2_fgm['tha_fgl_gsm'], dims=['time', 'components'], coords={'time':tha_l2_fgm['tha_fgl_time'].values, 'components':['x', 'y', 'z']}).plot.line(x='time')
plt.show()
# Works with matplotlib directly too
plt.plot(tha_l2_fgm['tha_fgl_time'], tha_l2_fgm['tha_fgl_gsm'])
plt.show()
Datetimes handling:
import pycdfpp
import os
# Due to an issue with pybind11 you have to force your timezone to UTC for
# datetime conversion (not necessary for numpy datetime64)
os.environ['TZ']='UTC'
mms2_fgm_srvy = pycdfpp.load("mms2_fgm_srvy_l2_20200201_v5.230.0.cdf")
# to convert any CDF variable holding any time type to python datetime:
epoch_dt = pycdfpp.to_datetime(mms2_fgm_srvy["Epoch"])
# same with numpy datetime64:
epoch_dt64 = pycdfpp.to_datetime64(mms2_fgm_srvy["Epoch"])
# note that using datetime64 is ~100x faster than datetime (~2ns/element on an average laptop)
Writing CDF files
Creating a basic CDF file:
import pycdfpp
import numpy as np
from datetime import datetime
cdf = pycdfpp.CDF()
cdf.add_attribute("some attribute", [[1,2,3], [datetime(2018,1,1), datetime(2018,1,2)], "hello\nworld"])
cdf.add_variable(f"some variable", values=np.ones((10),dtype=np.float64))
pycdfpp.save(cdf, "some_cdf.cdf")
C++
#include "cdf-io/cdf-io.hpp"
#include <iostream>
std::ostream& operator<<(std::ostream& os, const cdf::Variable::shape_t& shape)
{
os << "(";
for (auto i = 0; i < static_cast<int>(std::size(shape)) - 1; i++)
os << shape[i] << ',';
if (std::size(shape) >= 1)
os << shape[std::size(shape) - 1];
os << ")";
return os;
}
int main(int argc, char** argv)
{
auto path = std::string(DATA_PATH) + "/a_cdf.cdf";
// cdf::io::load returns a optional<CDF>
if (const auto my_cdf = cdf::io::load(path); my_cdf)
{
std::cout << "Attribute list:" << std::endl;
for (const auto& [name, attribute] : my_cdf->attributes)
{
std::cout << "\t" << name << std::endl;
}
std::cout << "Variable list:" << std::endl;
for (const auto& [name, variable] : my_cdf->variables)
{
std::cout << "\t" << name << " shape:" << variable.shape() << std::endl;
}
return 0;
}
return -1;
}
caveats
- NRV variables shape, in order to expose a consistent shape, PyCDFpp exposes the reccord count as first dimension and thus its value will be either 0 or 1 (0 mean empty variable).
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pycdfpp-0.6.4-pp310-pypy310_pp73-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5788494b63135f19866f5a77a25aa78b16341ce5207baf3d5af3c83ddc9086cd |
|
MD5 | cfa188c7cd7f0fcf642055ed9d2aa6df |
|
BLAKE2b-256 | 06359967d4c2682a47cfe4a084d17ed83b52ee390353e9b25ebc28b9c90a8286 |
Hashes for pycdfpp-0.6.4-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 85b5163f5c33fafc6cb7bcf2292b9608e3293c9377f0a50e5fe0a064527921bd |
|
MD5 | 5d149ba182cc0f27dda5ca3bf25ce1ec |
|
BLAKE2b-256 | c3f7daf45f5ae802a2af33a62714cf5cd5522521e2405c73bcf37cdaa0805a13 |
Hashes for pycdfpp-0.6.4-pp310-pypy310_pp73-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 66cba39ff49d179a5b447ff035cb5d717927306bb32e7e186336a7e54570d54a |
|
MD5 | 0c5861ed857e4e08f952651ad4d1d468 |
|
BLAKE2b-256 | 1fb1b850441401905e264af914e81ffd456d3d3c0463ecdb5a93f2789716f327 |
Hashes for pycdfpp-0.6.4-pp310-pypy310_pp73-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b74eddef7373b3d18ef5c5284feab28f4aa76209b40f5c2d37206f8fe912059b |
|
MD5 | 3ddc3d0d5031812f450e05523a3d3665 |
|
BLAKE2b-256 | 59d2cff30d4cc0a48887e6e1d86e4a23680ba7f7215230616ef0d63233ae2262 |
Hashes for pycdfpp-0.6.4-pp39-pypy39_pp73-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5e2c86c1d940812f9e4f4062aed1016b6f3f33a8d03e22600e9d91dfe45c9fd6 |
|
MD5 | d16607b0ec63c088ca19be8206609f5e |
|
BLAKE2b-256 | 68a7ec02220293cd6426ba50c301240095b584d54927c7452996ffbee4fa4c13 |
Hashes for pycdfpp-0.6.4-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 292d72d088fd843a03a5f3ed1d0870249362d70f635640bd95e996b2a58da71f |
|
MD5 | a09e314438f821d35c64f480ded5e9de |
|
BLAKE2b-256 | dd3dbabd1c48facd51189350d9109a2a6ae20a9db897260a6ec91b34e11a343a |
Hashes for pycdfpp-0.6.4-pp39-pypy39_pp73-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 115f5c6a9823312945e65559008405a3ad392b56a6fedd19e2b6f028a17f3de3 |
|
MD5 | e6558c08f68c8e53051cb44b0610879d |
|
BLAKE2b-256 | 2b1898aad21cf0390a9110706b60f6e157671c2d762183c73b4f7bdee19ddc5a |
Hashes for pycdfpp-0.6.4-pp39-pypy39_pp73-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4faac854350d65b2b6c1c5c80220906d65ca4933d38c9370753de6459ec885da |
|
MD5 | d830dc65cf158a951f9b6f265bd67d8d |
|
BLAKE2b-256 | 7bbcacc0047fd2d8a86863b39b57b155381974c34f4689713885722edb72f373 |
Hashes for pycdfpp-0.6.4-pp38-pypy38_pp73-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 562fe164102d84e037126b6fdd3a64a321d71de52d4b609d651c6d294bc2ca23 |
|
MD5 | bf957ab456db59a9d6dac3c04d3938e3 |
|
BLAKE2b-256 | c79fbfdd4377b9940661f840b9d8c1fde7abd96c76f4a2611e215405ca37842f |
Hashes for pycdfpp-0.6.4-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 36e04ddf5646dfcd9d73648ae8644bb8ad7c637ab01617c2e6e1f937383fb9e0 |
|
MD5 | 2ffe95d8b5b64d7e9ff7f5dc0c2d11f5 |
|
BLAKE2b-256 | 2720bbe8dfb59f69347f07552dd03c1cdba5452fb5493559e0a59dbe614f1b0c |
Hashes for pycdfpp-0.6.4-pp38-pypy38_pp73-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 355140abe2deb3d63c0ec60ed1692d864c9697a1bc36c17e0b538c43379d5775 |
|
MD5 | 2a96bb58303d3b46e04fa7d28d231eef |
|
BLAKE2b-256 | 85b26a8e4b93d8c9787d7555160ec895fde9258dd2f2a8d208edf2f8e2aff381 |
Hashes for pycdfpp-0.6.4-pp38-pypy38_pp73-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 23fb21e746acddc460229616636d087a8e681df474c1bc05ced6292ec91f8104 |
|
MD5 | d268733dd182c9a28fa8edff256246e7 |
|
BLAKE2b-256 | dc31369eb162cf9302933c80426c9a8e7f0f4dd0888775066d6dfd6c89c4a524 |
Hashes for pycdfpp-0.6.4-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1586f8a5b3e700bd7c16b60a44175fbf1017cf538ff465c2289ec330f8f8329b |
|
MD5 | 1c3022eddc08cc025d207e25d079315e |
|
BLAKE2b-256 | 0abaefe5620c3a99ba0a0ecee1aebb6d950ab8a7290506f54180fc126b1bfdf4 |
Hashes for pycdfpp-0.6.4-cp312-cp312-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d5b156f72daf3111985c4308cb78d3f0729cf02bb9a6d1ac1983325079a0348c |
|
MD5 | 8bfba104aae9bdca4d3c13d2ff866dce |
|
BLAKE2b-256 | 6e439d0e209ab91ce42f8a0691fd6dc31eec48853d4c6e0440d672c3904bd998 |
Hashes for pycdfpp-0.6.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a03437314fe37c453db73e20cae9d5d6db073447a4526664006ae3d531c6c596 |
|
MD5 | 514e8cabc2c990911980292689b4b096 |
|
BLAKE2b-256 | e71a24246a63922506ed611ba3d463b7f791d88d004fc57635f06c0a2267b9cb |
Hashes for pycdfpp-0.6.4-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6826a4e097e470d420202525d0bb4a79b8463b15b61d792d5fdf6b50ceea2fb7 |
|
MD5 | 56ebe4101b7f67d4ce9614fa54ce2368 |
|
BLAKE2b-256 | 6650710c0a0a857a55e63b33d6c2d6022c6fc1d0779b948b4e9d693b7e7e136a |
Hashes for pycdfpp-0.6.4-cp312-cp312-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9f7f99087fc1af455c9d5d929a2ebfb30751bdea62fbb262903cf1f8c5bbb1c1 |
|
MD5 | 0847258e0374c8bb84e96dc873fef470 |
|
BLAKE2b-256 | 7bb952e23b969ddc93e8c4f97381f147f06c9ae9275dabea4b1e4c930b90e83b |
Hashes for pycdfpp-0.6.4-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c600d8a46fd4c1a2324e8d8d6e0c945d1dba25c71d515419411e406ead3824fb |
|
MD5 | 197855b5dff2578182f1d95e29d613d6 |
|
BLAKE2b-256 | f0e57b40c6ac86577240d799975573b06b60dbc5ee284582e6be1bfdb58ef2eb |
Hashes for pycdfpp-0.6.4-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7239409f4aa379fda332568255a71ba1044d50423bc9fdf1699f192ab642d142 |
|
MD5 | b1b0b63b43954876eb3bdd350a5c5f90 |
|
BLAKE2b-256 | ef2c460c48751c9d65b5ba3b0b71a55d85c177210a15338e52a1e2872eb7f457 |
Hashes for pycdfpp-0.6.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bdbd3daad23361b83e453fc65346aea0ab57c7b9eb543be6c575eb0fd930c873 |
|
MD5 | 69d8fd8cb9cdfba0d28bd3748ee5b9c4 |
|
BLAKE2b-256 | 00a356c7545e40014e2cb07b675b21a052ac2d46782881e7e13bda028c40c5e4 |
Hashes for pycdfpp-0.6.4-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 016acc0b4d7a96a8376d928aa4512a2ec5e32aa54af623ea2f5717295034ad76 |
|
MD5 | 814e42a08cb41edb780f90106ce61930 |
|
BLAKE2b-256 | 58cc355f38de1de665ecc8df8f224bf19b6fa42d89ffc5bb671408c97c907ad0 |
Hashes for pycdfpp-0.6.4-cp311-cp311-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a34d4a9fb052dc9c98ca63b4e7a6caa2c5145c19e179757e3bed98d73a7b1fae |
|
MD5 | 2fec38d25f2a3f8da9e915dc116b17ee |
|
BLAKE2b-256 | a77a8c361830bfca535c6066b3d0e566d490b2f5a28453b6a3558a41741531ea |
Hashes for pycdfpp-0.6.4-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 486c3f217a7f58cff4ced4a6bf9ccdcbf7379d2f0d91d218aef8a53fe1039228 |
|
MD5 | 058437a04c7f9c828b7b87b4232aafc6 |
|
BLAKE2b-256 | 241b59d39ff96911086a1cf87d81a363db14b59cbeddd2e5e5d5cc98b2be046a |
Hashes for pycdfpp-0.6.4-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 07a43597bfdd05fa751902bc91ed7025a4b1480fc7cd3169e8f046cc8952e596 |
|
MD5 | 825d2ab6a817c7185b604a2acfc6ac5d |
|
BLAKE2b-256 | f7310569d6de34e0fad626979246c60dfcdc2e29598e3f971c1946475054aaf9 |
Hashes for pycdfpp-0.6.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1e47f0c127cf475016d4ab635c68519ff7f200f4c325b23e8489540e0dc057ae |
|
MD5 | e2238f9036f75b06db0664938f40ed5c |
|
BLAKE2b-256 | d64dff7bc3d27d8831b85b2fab16d64812b281821dddb819358f71fefe3c12ee |
Hashes for pycdfpp-0.6.4-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 755d263bcce612d099c3222be52e8a4d6f373716b8388433242d620ba765f23e |
|
MD5 | d0dcdcaf4c4ae7470b939c4e9fa1d055 |
|
BLAKE2b-256 | 5e6ab51a1b69f1136d8e82c514879796c4d9ceb7403e0742c45ecadcbaebd3cb |
Hashes for pycdfpp-0.6.4-cp310-cp310-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fbd920c57f7a9437de27ede1e793715544e301adfb0b9c7b79375270e7266794 |
|
MD5 | fdb66ac82e13147fb2204760fbaee854 |
|
BLAKE2b-256 | 92d2bcb962dcf16bdcfdd0e5e200518f9052e7eb4e24fab173bed17b972aef2e |
Hashes for pycdfpp-0.6.4-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 59d6f9fb518b17b93b945cf704eeb10cd6c4fea7ad8217bcb292712f20be665d |
|
MD5 | a599e904631f1e46fb39697d807b742b |
|
BLAKE2b-256 | 074456eef04fe3c2d97c5fb70b180b55d20a6d5b6ad42b2e0f4b75e86cc5f6e6 |
Hashes for pycdfpp-0.6.4-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8be7816defc2997de39e773c49386ebe48ca3d49dd19da52db1ce0e6c66b11e6 |
|
MD5 | e94b3b01fcce0f9fb2a39037f7428f00 |
|
BLAKE2b-256 | bdf32d9dc2559c3560dc1c9d7f42fb5d4ec8508ce4b6830feac352f6114d7db1 |
Hashes for pycdfpp-0.6.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 364ab2b7dbf4ac6ae67bfc2433d2342dbd05fc3c3b8569142e0a0409164159db |
|
MD5 | 0a0975caeed0999f48a206b506016858 |
|
BLAKE2b-256 | 26fe150376e7521ded92e8f5e3de2a75bfadf31f3ebe91cc64170559e4adc3fd |
Hashes for pycdfpp-0.6.4-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eaca6f26ed35439c8c6f94c32ea6e96baa19092349feb0c4c57279551c2446ed |
|
MD5 | 51895a8f35125895be3a5abde20e860b |
|
BLAKE2b-256 | 1b3cd33730659e50e1d73dfa8f0b54e900ce36571883167873dd6d2ecaa70122 |
Hashes for pycdfpp-0.6.4-cp39-cp39-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f4c87f4d51be0bd6aaeb2937896094387b4aba28b1a524e0ba2c67f8912e2bdd |
|
MD5 | 9723b7e704fd8242aa0650a463d4eb10 |
|
BLAKE2b-256 | 9f72e4098ad5b37a06aaa02c1ebc9d44d6dc3e9cd2a1beda748b0d521c2339b6 |
Hashes for pycdfpp-0.6.4-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 76d5745e1aa097cb876ec0e4582c01ad8c08fa3cd38a332d12303f5a546adb8f |
|
MD5 | ffad71bd2ab74b2b194bb63ed407abf4 |
|
BLAKE2b-256 | beee6b90264a6a5ea3fe68d57ebae3a31a6e6d10c0d9ad7eeab4a2a62c29e754 |
Hashes for pycdfpp-0.6.4-cp38-cp38-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d7473df6a2a1d80e4db4ca6cbe6ce70f4ea8c7ec43d517fdd6bde8489997c7ff |
|
MD5 | b79a300415e663cdbc8e4a9b79600dbf |
|
BLAKE2b-256 | 31021dcd8bc6be9a46c59166a498d5b740de8d1128b008be46911ccd91703310 |
Hashes for pycdfpp-0.6.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ab2e0c8fa3e886f5302b88b476bf00c59c7930f3765df828aa86b25ac453262d |
|
MD5 | 1d8200ba418b58601eb019cd56695c73 |
|
BLAKE2b-256 | c40949105087921f0c572e949e6987c02412649f864bf56272a8f7386056f3d1 |
Hashes for pycdfpp-0.6.4-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0b8bf8bd09221fbc952017ede9137aac7b5c833550f7da8e7791c072078d344c |
|
MD5 | da931ad23859029a7f26811107523b90 |
|
BLAKE2b-256 | 0442d9d20ffe8f71141c7daaea3aa0181b7bb9a46f251b5209330b4e7303a5f7 |
Hashes for pycdfpp-0.6.4-cp38-cp38-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 869ff7f0f41fe0815812fce1614714d4625d50d1aa12750824375ed01241057c |
|
MD5 | 94dbd6f20557c50a82e48e36aa691d67 |
|
BLAKE2b-256 | a33cc9533ffbc1292df18bd41ee0f6256be92d08ceb712d62386fc78569d6bc5 |