Common Data Model reader and mapper toolbox
Verified details
These details have been verified by PyPIProject links
GitHub Statistics
Maintainers
Project description
Versions |
|
Documentation and Support |
|
Open Source |
|
|
|
Coding Standards |
|
|
|
Development Status |
|
Funding |
|
|
The cdm_reader_mapper toolbox is a python tool designed for:
to read data files compliant with a user specified data model
to map observed variables and its associated metadata from a data model or models combination to the C3S CDS Common Data Model (CDM) format
detect and flag or remove duplicated observations
It was developed to read the IMMA (International Maritime Meteorological Archive) data format, but it has been enhanced to account for meteorological data formats in the case of:
Data that is stored in a human-readable manner: “ASCII” format.
Data is stored in a Network Common Data Format: “NetCDF” format.
Data that is organized in single line reports
Reports that have a coherent internal structure and can be modelised.
Reports that have a fixed width or field delimited types
Reports that can be organized in sections, in which case each section can be of different types (fixed width of delimited)
Installation
Installation using pip
You can install the package directly from pip:
pip install cdm_reader_mapperIf you want to contribute, we recommend cloning the repository and installing the package in development mode, e.g.
git clone https://github.com/glamod/cdm_reader_mapper
cd cdm_reader_mapper
pip install -e .This will install the package but you can still edit it and you don’t need the package in your PYTHONPATH
Installation using uv
You can install the package using uv package manager, this will add the library to your active environment:
uv add cdm_reader_mapperTo develop the package using uv, the following will create a virtual environment, uv defaults to .venv:
git clone https://github.com/glamod/cdm_reader_mapper
cd cdm_reader_mapper
uv venv --python 3.12 # Create an environment with the recommended python version
source .venv/bin/activate # Load the virtual environment (for bash or zsh)
uv syncDocumentation
The official documentation is at https://cdm-reader-mapper.readthedocs.io/
How to make the most of cdm_reader_mapper:
Logging
By default, cdm_reader_mapper outputs logging information to stdout. To tell cdm_reader_mapper to output logs to a file, set the CDM_LOG_FILE environment variable before loading cdm_reader_mapper.
import os
os.environ["CDM_LOG_FILE"] = "log_file.log"
import cdm_reader_mapper as cdmThis will set the file log_file.log as the output for all logging information from cdm_reader_mapper, including the initial logging on loading of the package.
Run a test
Read imma data with the cdm_reader_mapper.read_mdf function:
from cdm_reader_mapper import read_mdf
from cdm_reader_mapper.data import test_data
imodel = "icoads_r300_d701"
data = test_data.test_icoads_r300_d701.get("source")
db_imma = read(filepath, imodel=imodel, sections=["core", "c1", "c98"])Map this data to a CDM build for the same deck (in this case deck 704: US Marine Metereological Journal collection of data):
db_cdm = db_imma.map_model(
data_raw,
imodel=imodel,
log_level="DEBUG",
)Detect and flag duplicated observations:
imma_bundle.duplicate_check(inplace=True)
db_flagged = imma_bundle.flag_duplicates()
df_flagged = db_flagged.dataFor more details on how to use the cdm_reader_mapper toolbox see the following jupyter example notebooks.
Contributing to cdm_reader_mapper
If you’re interested in participating in the development of cdm_reader_mapper by suggesting new features, new indices or report bugs, please leave us a message on the issue tracker.
If you would like to contribute code or documentation (which is greatly appreciated!), check out the Contributing Guidelines before you begin!
How to cite this library
If you wish to cite glamod-marine-processing in a research publication, we kindly ask that you refer to Zenodo: https://zenodo.org/records/14135493.
License
This is free software: you can redistribute it and/or modify it under the terms of the Apache License 2.0. A copy of this license is provided in the code repository (LICENSE).
Credits
cdm_reader_mapper development is funded through Copernicus Climate Change Service (C3S).
Furthermore, acknowledgments go to National Oceanography Centre (NOC) and UK Marine and Climate Advisory Service (UKMCAS).
This package was created with Cookiecutter and the audreyfeldroy/cookiecutter-pypackage project template.
