A modern C++ header only cdf library
Project description
Python packages
Linux x86_64 | Windows x86_64 | MacOs x86_64 | MacOs ARM64 |
---|---|---|---|
Unit Tests
Linux x86_64 | Windows x86_64 | MacOs x86_64 |
---|---|---|
CDFpp (CDF++)
A NASA's CDF modern C++ library. This is not a C++ wrapper but a full C++ implementation. Why? CDF files are still used for space physics missions but few implementations are available. The main one is NASA's C implementation available here but it lacks multi-threads support (global shared state), has an old C interface and has a license which isn't compatible with most Linux distributions policy. There are also Java and Python implementations which are not usable in C++.
List of features and roadmap:
- CDF reading
- read files from cdf version 2.2 to 3.x
- read uncompressed file headers
- read uncompressed attributes
- read uncompressed variables
- read variable attributes
- loads cdf files from memory (std::vector or char*)
- handles both row and column major files
- read variables with nested VXRs
- read compressed files (GZip, RLE)
- read compressed variables (GZip, RLE)
- read UTF-8 encoded files
- read ISO 8859-1(Latin-1) encoded files (converts to UTF-8 on the fly)
- variables values lazy loading
- decode DEC's floating point encoding (Itanium, ALPHA and VAX)
- pad values
- CDF writing
- write uncompressed headers
- write uncompressed attributes
- write uncompressed variables
- write compressed variables
- write compressed files
- pad values
- General features
- uses libdeflate for faster GZip decompression
- highly optimized CDF reads (up to ~4GB/s read speed from disk)
- handle leap seconds
- Python wrappers
- Documentation
- Examples (see below)
- Benchmarks
If you want to understand how it works, how to use the code or what works, you may have to read tests.
Installing
From PyPi
python3 -m pip install --user pycdfpp
From sources
meson build
cd build
ninja
sudo ninja install
Or if youl want to build a Python wheel:
python -m build .
# resulting wheel will be located into dist folder
Basic usage
Python
Reading CDF files
Basic example from a local file:
import pycdfpp
cdf = pycdfpp.load("some_cdf.cdf")
cdf_var_data = cdf["var_name"].values #builds a numpy view or a list of strings
attribute_name_first_value = cdf.attributes['attribute_name'][0]
Note that you can also load in memory files:
import pycdfpp
import requests
import matplotlib.pyplot as plt
tha_l2_fgm = pycdfpp.load(requests.get("https://spdf.gsfc.nasa.gov/pub/data/themis/tha/l2/fgm/2016/tha_l2_fgm_20160101_v01.cdf").content)
plt.plot(tha_l2_fgm["tha_fgl_gsm"])
plt.show()
Buffer protocol support:
import pycdfpp
import requests
import xarray as xr
import matplotlib.pyplot as plt
tha_l2_fgm = pycdfpp.load(requests.get("https://spdf.gsfc.nasa.gov/pub/data/themis/tha/l2/fgm/2016/tha_l2_fgm_20160101_v01.cdf").content)
xr.DataArray(tha_l2_fgm['tha_fgl_gsm'], dims=['time', 'components'], coords={'time':tha_l2_fgm['tha_fgl_time'].values, 'components':['x', 'y', 'z']}).plot.line(x='time')
plt.show()
# Works with matplotlib directly too
plt.plot(tha_l2_fgm['tha_fgl_time'], tha_l2_fgm['tha_fgl_gsm'])
plt.show()
Datetimes handling:
import pycdfpp
import os
# Due to an issue with pybind11 you have to force your timezone to UTC for
# datetime conversion (not necessary for numpy datetime64)
os.environ['TZ']='UTC'
mms2_fgm_srvy = pycdfpp.load("mms2_fgm_srvy_l2_20200201_v5.230.0.cdf")
# to convert any CDF variable holding any time type to python datetime:
epoch_dt = pycdfpp.to_datetime(mms2_fgm_srvy["Epoch"])
# same with numpy datetime64:
epoch_dt64 = pycdfpp.to_datetime64(mms2_fgm_srvy["Epoch"])
# note that using datetime64 is ~100x faster than datetime (~2ns/element on an average laptop)
Writing CDF files
Creating a basic CDF file:
import pycdfpp
import numpy as np
from datetime import datetime
cdf = pycdfpp.CDF()
cdf.add_attribute("some attribute", [[1,2,3], [datetime(2018,1,1), datetime(2018,1,2)], "hello\nworld"])
cdf.add_variable(f"some variable", values=np.ones((10),dtype=np.float64))
pycdfpp.save(cdf, "some_cdf.cdf")
C++
#include "cdf-io/cdf-io.hpp"
#include <iostream>
std::ostream& operator<<(std::ostream& os, const cdf::Variable::shape_t& shape)
{
os << "(";
for (auto i = 0; i < static_cast<int>(std::size(shape)) - 1; i++)
os << shape[i] << ',';
if (std::size(shape) >= 1)
os << shape[std::size(shape) - 1];
os << ")";
return os;
}
int main(int argc, char** argv)
{
auto path = std::string(DATA_PATH) + "/a_cdf.cdf";
// cdf::io::load returns a optional<CDF>
if (const auto my_cdf = cdf::io::load(path); my_cdf)
{
std::cout << "Attribute list:" << std::endl;
for (const auto& [name, attribute] : my_cdf->attributes)
{
std::cout << "\t" << name << std::endl;
}
std::cout << "Variable list:" << std::endl;
for (const auto& [name, variable] : my_cdf->variables)
{
std::cout << "\t" << name << " shape:" << variable.shape() << std::endl;
}
return 0;
}
return -1;
}
caveats
- NRV variables shape, in order to expose a consistent shape, PyCDFpp exposes the reccord count as first dimension and thus its value will be either 0 or 1 (0 mean empty variable).
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pycdfpp-0.7.0-pp310-pypy310_pp73-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1b9ef3b6f46ffbc8d1a5f75b15be702b952c3fccbad972207e4618993e3690f1 |
|
MD5 | 8fa1f7324e9ed9931950f43e639c6030 |
|
BLAKE2b-256 | 467252a538e7897ad80253ae3d91793f8da2bfc81c7f5695f54f7d54a47ce137 |
Hashes for pycdfpp-0.7.0-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cdce4463ca46ce0a1d5fadf70397fa97abdc3d5e2cae9e095e9d7ee79508afb1 |
|
MD5 | be7211a0fc191a8930c9dab92ea91853 |
|
BLAKE2b-256 | f740167070b420ac00a200c552eb02b7579b44aa808927c2a0d10e0a2c081de0 |
Hashes for pycdfpp-0.7.0-pp310-pypy310_pp73-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5bfedc3d20068bc684dfa870dff225870646815f8d48e361e7d793a207435b8b |
|
MD5 | 2db3d96d44e9edfbc9eea229c7c9432c |
|
BLAKE2b-256 | 16c81e20e96dbfae5ecea97c500b505d76f8cc721ece9036b0589a2450e4d4c4 |
Hashes for pycdfpp-0.7.0-pp310-pypy310_pp73-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 744cf4b977818f926957fec882675452698a5a4d3da5176c3e80c0c1754f0161 |
|
MD5 | 1144db8e7081dac2d3c2940e8ca056dc |
|
BLAKE2b-256 | 4bc55fea23fedaa4dc697a7ebdcac063fd8b9feae73b91fa0d514348b5690971 |
Hashes for pycdfpp-0.7.0-pp39-pypy39_pp73-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | da5120cf4b93bf2afe90441e582b8394ed4265ed439ae309fa88b37048818e54 |
|
MD5 | 9691a7cb705ccf854557de35187738d4 |
|
BLAKE2b-256 | 85e2925a65ac3a53ee7ac616feae6339b85e02c047cd189865e412ffc9f55773 |
Hashes for pycdfpp-0.7.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ecfdab5fdd8e319b9a8a0e6aa5fb13c56a3b2cc7a8faeb15744b2e2dd6f16231 |
|
MD5 | b270dd14919217a91931fc70de1fcba3 |
|
BLAKE2b-256 | 9df3600989df03f6c08b7a93659e608160ece70dd1d40ed5975311536569fb43 |
Hashes for pycdfpp-0.7.0-pp39-pypy39_pp73-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5d6890a95243d322c4fe8a99096ed6d43efe7f684c9310d873cc62d879c9a01c |
|
MD5 | aff498140c616a02c1513689a1c1ebf8 |
|
BLAKE2b-256 | 1c5f77c25eebcef076d791ab24d384cc04f7e9509a642f9e98e826658342139d |
Hashes for pycdfpp-0.7.0-pp39-pypy39_pp73-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3578559a3882b07793d4e225086d8724c1340e8b63fa6a81322bd6376eb5266a |
|
MD5 | 70840f52ca9fecfbdb89dcec90f962e5 |
|
BLAKE2b-256 | 3578e639b96a0bcba17b39edf9d7270fcd2be8962146b8e9a0ae08d3b8d77278 |
Hashes for pycdfpp-0.7.0-pp38-pypy38_pp73-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6ca2cc4e8bebae1c85925c16542c1f04827e22d6966dede7240890b3d4f2b883 |
|
MD5 | b46d6ce2043fb70fb96fbc6a225fda22 |
|
BLAKE2b-256 | 1d5a60bf50edd3c73f6c27f7cecb6db93165706b26af00b24356fb9dcc70f441 |
Hashes for pycdfpp-0.7.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e05545a4078f574325437edfd745905b5763edda35fa34e46e3e1b40d80b9341 |
|
MD5 | c2347a5f6595b99564a3f7a7ca5f7b3d |
|
BLAKE2b-256 | 7d92f827aa3890b0e68fe78871a6f69aa341c741a9031b1a4ef447dbc0719fc8 |
Hashes for pycdfpp-0.7.0-pp38-pypy38_pp73-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 22959c5449b0c0823f5df3b0cb9456286bec9d6a6fc5e98464dc1fdc39698007 |
|
MD5 | ed6e0f763c1dab12dcd74961179c822f |
|
BLAKE2b-256 | ed8726c6de7016645d30fc4ef2af0e0ab100090685b3527db58a1fc75a117f8c |
Hashes for pycdfpp-0.7.0-pp38-pypy38_pp73-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 508ba0517801a6b6205bd05922eba25e7ff0a69e88da898f5bf2e71b451c390d |
|
MD5 | b4c1f7e3da141325f79f6fa52dfb3fdf |
|
BLAKE2b-256 | 06941f207cf0672d91d22c5104acacd3b2bce972bc17eed0e23b6008622495cb |
Hashes for pycdfpp-0.7.0-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f21c4503a610a5fe1a6e333c34dc43fdd04a0a1426e96742b057f7554b239ee5 |
|
MD5 | 014d9b0001faa08a5aac11477ddb626d |
|
BLAKE2b-256 | 685efef03c91d90e6d4009ea33869f18ddb056f09e78566aba94ba06a7272287 |
Hashes for pycdfpp-0.7.0-cp312-cp312-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cd5cd189715d69d06f23099e02bfc97f56680aa58e5bf5e7f34a829e980e138f |
|
MD5 | eed2007c4382760ce716040ca3b04489 |
|
BLAKE2b-256 | 20a2bf51f2153d2c24f2b3b8d43d82733ee5e0a83b26559a110789c6be2c3205 |
Hashes for pycdfpp-0.7.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a6695d73b02a3dbee6a9c3d21bf1c74a08f7d4c464b503288a992a2dfab8061d |
|
MD5 | 33cef5c64257ece75d5c91b453f3394c |
|
BLAKE2b-256 | 31be7769d7515a09bf18eae58c8660e8d06533ea1a8527c5a70ae768f4bf54c7 |
Hashes for pycdfpp-0.7.0-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9b4d04991312877d1d960e798bcdfa9a5b950bcb51f9ca30a227425be1d9d3c5 |
|
MD5 | 816c60b29317b3c1ef9017d3abbdd639 |
|
BLAKE2b-256 | 9ce60c63d84e83ea6da29685cfd1a6de0992920b46b3e5fe183cd9352d175aa8 |
Hashes for pycdfpp-0.7.0-cp312-cp312-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fa37ba32d2cf73642b71e4379b55fb487f7fc7def0559f8d52839882923b5089 |
|
MD5 | f0de678da64ec90e460ac1303a08d032 |
|
BLAKE2b-256 | 2722da687dab277fc979c74ddc4fc66224e5684f654b4e696b959fa28d46e5c2 |
Hashes for pycdfpp-0.7.0-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3182546743aed21039a178daa0642da8bb821df14331e4f46cbedf18ec56b40c |
|
MD5 | 7d5d4183e91ba7f7e4760b7d705013d6 |
|
BLAKE2b-256 | dd657623e781039778bc6a6ceae73288e3d49e8bb9de7ec8bfa1e0a52c639a75 |
Hashes for pycdfpp-0.7.0-cp311-cp311-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c186e8f89df3c94dece419973f8c6f7c012c6fce70fdd410a07a35054043a2f9 |
|
MD5 | 50c0acd0be05167474e7db5dbbda48d6 |
|
BLAKE2b-256 | 721d04de90245bdaf90869bb5cb116218618907f134a87f6eefc1e5cb76833b9 |
Hashes for pycdfpp-0.7.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c80ef53d9c250b3dd8bbccf72357d33e9b8507da4640bbb9fd8f09f56f3e86f8 |
|
MD5 | 0a85524b468f1cdb7a4512713a4d82b7 |
|
BLAKE2b-256 | 081a6bb8ba811010be141e9b0100b8aa30a81806fbc6f2691ad4e94728511773 |
Hashes for pycdfpp-0.7.0-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3ae93085912c0aaa32f5754a1f4a1860832d38eb487a52559f4c74ab80a8f79b |
|
MD5 | ae243e5a402e5f97d05135ce28c354c5 |
|
BLAKE2b-256 | 10b83f7fc3a89d363c49087369a33671edda631d1ca1e79235c1fa37d736a18d |
Hashes for pycdfpp-0.7.0-cp311-cp311-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c1c363e6f415191dbf5fd8fcb0c936cde5ce2018e97364c53b8bcd6ae6dfc83b |
|
MD5 | 02cd4bd37fe1c2de49b92d5389a9de96 |
|
BLAKE2b-256 | c3f96bb87502e0194ce1a92e1dd0eea0f8e0ba2632d5452bf7f7414e8ad25052 |
Hashes for pycdfpp-0.7.0-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 68362e21aa677421547245f1f9e21ddf8d8f145e5287702afe20dae590b58226 |
|
MD5 | 68bd02f76f95d32f1d64e86de2c19918 |
|
BLAKE2b-256 | 56dff36acacb07edde9d3bcd6c24794296298686bd1fd4ee18850f7fb7c67f77 |
Hashes for pycdfpp-0.7.0-cp310-cp310-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cf9ecb9f01e877c62f270915c642068d95e8ff8097209f46438ad998c423b547 |
|
MD5 | be1e0e49b8b51fff535c686ed81119b4 |
|
BLAKE2b-256 | 32875343b607b776b615fed11a0f7d3ca4a5d0775421523555a807e8e8e6e9ab |
Hashes for pycdfpp-0.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aaa6b583e13f60085fdd3e6c05c411b6c38a1217c94934ae156797d5cd7346ad |
|
MD5 | 05c9260bac10ba94f66406371824425f |
|
BLAKE2b-256 | 03d3692c9c22d28cad951cc50ac876ceb15d4c18a98e1f7fabdf6aec6eea5303 |
Hashes for pycdfpp-0.7.0-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b011a0559d38c6b13aa4078b685bfe036b509649a0049614c1cae4b23b848073 |
|
MD5 | 7fa2ce809b5c43fe51ade7bbc4f924f8 |
|
BLAKE2b-256 | aa9e8cd45708f8db424250d9c95b83b93829bcad6964fcb7e19a1a35d60d60e1 |
Hashes for pycdfpp-0.7.0-cp310-cp310-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a812d5c3d5b420cda7ed791854f3d659691281aad05bd7936db0f0ce1b179fb5 |
|
MD5 | decfcbf563d4d7c14d259dd95e8ab095 |
|
BLAKE2b-256 | dbcf4cf9ba6683992545df50bb7579c6b8755c0df8a33a676f50e297ea9a8fd8 |
Hashes for pycdfpp-0.7.0-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8f290268003751a71a97d2d5ef270f088b020191b4c41758dc557f00128d998d |
|
MD5 | 607cb45206f2b83fa8352696de9b7d46 |
|
BLAKE2b-256 | f9b4157b3f6e171bc7552aff5ae5f35c00e3fc65d1cfa5e1f91a0b96c2a45d6c |
Hashes for pycdfpp-0.7.0-cp39-cp39-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 97820916d683737b6be9424f052e02ee42b445deddffb7567a6d8d9bd483b8d7 |
|
MD5 | 7c3504f48301f742080c6d266c284ca6 |
|
BLAKE2b-256 | c383fd59e58c63abb41f60b3c4f08547378c8c0eca5b5af9f3e1e942de8011cd |
Hashes for pycdfpp-0.7.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 434cc9242661efd9cd34bc4e4fce4402b597cd8f9ddbe0efb8c87b44db5a771e |
|
MD5 | 6d26bcc7c269d292993ec22ce5bd6054 |
|
BLAKE2b-256 | a8d79747a8eb4177d4a7e86c2ab117d6e47de8179a5a8fa819d92973a781d933 |
Hashes for pycdfpp-0.7.0-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 881bd3ceb906d0ae123146176611ede26a953cea5269f1547a4ec200eb82f73f |
|
MD5 | c47ae77265314802568319cc0d990e44 |
|
BLAKE2b-256 | c18545b9a0896d609186c96906a4d4ababb148d6065c64bf143320d1279dc630 |
Hashes for pycdfpp-0.7.0-cp39-cp39-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ddea55ebdeeed795f77760c183487b69ff146c7624356c5b4a60fbc31602fd1b |
|
MD5 | a4117ccfb32833419bbf4f584cbba178 |
|
BLAKE2b-256 | 91c04efca8d454bb1c33d213507cd1f519eda2995942fb0e69a34812274e6008 |
Hashes for pycdfpp-0.7.0-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b6699c665000b6b2d207e2ff61aefcf8f705ac874b3fd86accd2ac7ecbc9311b |
|
MD5 | 0de5da6b8fb28cb0bc93063d4d3f82a4 |
|
BLAKE2b-256 | 0d0c52f700eb753cb2006168d7d3dd332ca6d109f7d59fc87248c1fa7f6d3bdb |
Hashes for pycdfpp-0.7.0-cp38-cp38-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f50ab7e6df19df580a0c41a6e049a51a9f8b4f2627f96d2d90b2f6ffc4eda65a |
|
MD5 | 32333912f8d9e19f1fb06e590ff27f51 |
|
BLAKE2b-256 | 9efd123aa063ea500ad22f9f7b6dc90b493f3c17584f6a14b82b4abc1a43277a |
Hashes for pycdfpp-0.7.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a9026c0a5c114ebc899676132e8db91a3cd621965eba6025aaf13ab35085f643 |
|
MD5 | af4d4dbd4001f075812d51d780b35b41 |
|
BLAKE2b-256 | 283ddb9ffe9e0c346c7b1836b26086bcde30736434088860fe27ebd685db558d |
Hashes for pycdfpp-0.7.0-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 99578277f7404bd81d1ac5471fb83e47212439ab2b681fd2cf0248d4c473d775 |
|
MD5 | 028b8d25afab83dd2ef37e79bd3d23d6 |
|
BLAKE2b-256 | eb8d59389ddd2de157eb878ec1e301e0f099bc4fc9c2407e6ed56189b72b2255 |
Hashes for pycdfpp-0.7.0-cp38-cp38-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f0915fc518e4570a983879caf8fc44a8d85b1ed5a8a4dacfae1d51bd5bac137d |
|
MD5 | e4efecd8e80fe9456b9639056a13664e |
|
BLAKE2b-256 | b6262d32949cc536d88ac27221e14f1998ca2493e6212df33a604860f99bfa18 |