cotede 0.17.0

Why use CoTeDe

CoTeDe can apply different quality control procedures:

• The default GTSPP, EGOOS or Argo procedures;

• A custom set of tests, including user defined thresholds;

• A novel approach based on Anomaly Detection, described by Castelao 2015;

• Fuzzy Logic approach proposed by Morello et.al. 2014;

• A complete fuzzy logic procedure, with traditional deffuzification, proposed by Castelão 2016;

Process multiple files in parallel, ideal for large datasets.

Export output, original data plus flags, into netCDF files following OCEANSites data structure.

Quick howto

To evaluate the records of a profile:

import cotede.qc

pqc = cotede.qc.fProfileQC(‘example.cnv’)

To see the temperature records of the primary sensor:

pqc[‘temperature’]

To see the flags of all tests applied on the secondary sensor of salinity:

pqc.flags[‘salinity2’]

To evaluate a full set of profiles at once, like all profiles from a cruise, use the class ProfileQCCollection, like:

dataset = ProfileQCCollection(‘/path/to/data/’, inputpattern=”.*.cnv”)

dataset.flags[‘temperature’].keys()

Check the notebooks galery for more examples and functionalities: http://nbviewer.ipython.org/github/castelao/CoTeDe/tree/master/docs/notebooks/

Documentation

http://cotede.readthedocs.org

Why the name CoTeDe?

Since NOAA I wanted to combine the multiple tests, but I didn’t really knew how to do that. In 2011 I learned the anomaly detection technique, but I only formalize the procedure in 2013, when I spent few months in Toulouse. The full name of this package is CoTe De l’eau, which I understand as something near to “rating the water”. The short name is cotede, to make easier for the users to remember, since it is the quality control of COnductivity TEmperature and DEpth (cotede). The french name is a kind of tribute to the great time that I spent in France with Bia and the croissants that were converted in code lines.

0.17 - Mar, 2016

• Implementing fuzzy procedures inside CoTeDe, thus removing dependency on scikit-fuzzy. scikit-fuzzy is broken, hence compromising tests and development of CoTeDe.

0.16 - Mar, 2016

• Using external package OceansDB to handle climatologies and bathymetry.

0.15 - Dec, 2015

• Moved procedures to handle climatology to external standalone packages.

0.14 - Aug, 2015

• Interface for human calibration of anomaly detection

• Implemented fuzzy logic criteria

0.13 - July, 2015

• Major improvements in the anomaly detection submodule

• Partial support to thermosalinographs (TSG)

• Working on WOA test to generalize for profiles and tracks

• Adding .json to default QC configuration filenames

• Moved load_cfg from qc to utils

0.12

Since 0.9 some of the most important changes.

• Following CF vocabulary for variables names (PRES, TEMP, PSAL…)

• Partial support to ARGO profiles

• Added density invertion test

• Included haversine to avoid dependency on MAUD.

• tox and travis support.

0.9 - Dec, 2013

• Going public

0.7.3

• Creating fProfileQC()

0.5.4 - Nov, 2013

• Including Tukey53H test

0.5.0

• Implemented ProfileQCCollection

0.4 - Sep, 2013

• gradient and spike tests with depth conditional thresholds

• CruiseQC

• Use default threshold values for the QC tests.

0.1 - May 24, 2013

• Initial release.

QC_ML - 2011

• QC_ML, a machine learning approach to quality control hydrographic data, the initial prototype of CoTeDe. I refactored the system I developed to quality control TSG, to evaluate the PIRATA’s CTD stations for INPE. At this point I migrated from my personal Subversion server to Bitbucket, and I lost the history and logs before this point.

2006

• A system to automaticaly quality control TSG data on realtime for AOML-NOAA. The data was handled in a PostgreSQL database, and only the traditional tests were applied, i.e. a sequence of binary tests (spike, gradient, valid position …).