Skip to main content

Data Assimilation with Python: a Package for Experimental Research.

Project description

DAPPER is a set of templates for benchmarking the performance of data assimilation (DA) methods. The tests provide experimental support and guidance for new developments in DA. Example diagnostics:

EnKF - Lorenz'63

The typical set-up is a twin experiment, where you

  • specify a
    • dynamic model*
    • observational model*
  • use these to generate a synthetic
    • "truth"
    • and observations thereof*
  • assess how different DA methods perform in estimating the truth, given the above starred (*) items.

DAPPER enables the numerical investigation of DA methods through a variety of typical test cases and statistics. It (a) reproduces numerical benchmarks results reported in the literature, and (b) facilitates comparative studies, thus promoting the (a) reliability and (b) relevance of the results. DAPPER is (c) open source, written in Python, and (d) focuses on readability; this promotes the (c) reproduction and (d) dissemination of the underlying science, and makes it easy to adapt and extend. In summary, it is well suited for teaching and fundamental DA research. Also see its drawbacks.

Installation

  1. Prerequisite: python>=3.6 (suggest setting it up with anaconda).
    Execute python -V (uppercase V) in a terminal to assert that the version is 3.6 or higher.
  2. Install: Download and extract (or git clone) DAPPER, cd into the resulting folder, and pip install -e .
  3. Test: python example_1.py

Step 2 can be replaced by pip install da-dapper but this is not recommended since this hides away DAPPER as a library in your python path.

Getting started

Read, run, and understand the scripts example_{1,2,3}.py. Then, get familiar with the code.

The docs provide processed docstrings, but are far from complete.

Alternatively, see DA-tutorials for an intro to DA.

Methods

References

Method Literature reproduced
EnKF 1 Sak08, Hot15
EnKF-N Boc12, Boc15
EnKS, EnRTS Raa16b
iEnKS / iEnKF / EnRML / ES-MDA 2 Sak12, Boc12, Boc14
LETKF, local & serial EAKF Boc11
Sqrt. model noise methods Raa15
Particle filter (bootstrap) 3 Boc10
Optimal/implicit Particle filter 3 Boc10
NETF Töd15, Wil16
Rank histogram filter (RHF) And10
4D-Var
3D-Var
Extended KF
Optimal interpolation
Climatology

1: Stochastic, DEnKF (i.e. half-update), ETKF (i.e. sym. sqrt.). Serial forms are also available.
Tuned with inflation and "random, orthogonal rotations".
2: Also supports the bundle version, and "EnKF-N"-type inflation.
3: Resampling: multinomial (including systematic/universal and residual).
The particle filter is tuned with "effective-N monitoring", "regularization/jittering" strength, and more.

Models

Model Linear? Phys.dim. State len # Lyap≥0 Implementer
Lin. Advect. Yes 1d 1000 * 51 Evensen/Raanes
DoublePendulum No 0d 4 2 Matplotlib/Raanes
LotkaVolterra No 0d 5 * 1 Wikipedia/Raanes
Lorenz63 No 0d 3 2 Sakov
Lorenz84 No 0d 3 2 Raanes
Lorenz95 No 1d 40 * 13 Raanes
LorenzUV No 2x 1d 256 + 8 * ≈60 Raanes
Quasi-Geost No 2d 129²≈17k ≈140 Sakov

*: flexible; set as necessary

Other reproductions

As mentioned above, DAPPER reproduces literature results. There are also plenty of results in the literature that DAPPER does not reproduce. Typically, this means that the published results are incorrect.

A list of experimental settings that can be compared with literature papers can be obtained using gnu's find:

		$ find . -iname "[a-z]*[0-9].py" | grep mods

Some of these files contain settings that have been used in several papers.

Alternative projects

DAPPER is aimed at research and teaching (see discussion on top). Example of limitations:

  • It is not suited for very big models (>60k unknowns).
  • Time-dependent error covariances and changes in lengths of state/obs (although the Dyn and Obs models may otherwise be time-dependent).
  • Non-uniform time sequences not fully supported.

Also, DAPPER comes with no guarantees/support. Therefore, if you have an operational (real-world) application, such as WRF, you should look into one of the alternatives, sorted by approximate project size.

Name Developers Purpose (approximately)
DART NCAR Operational, general
PDAF AWI Operational, general
JEDI JCSDA (NOAA, NASA, ++) Operational, general (in develpmt?)
ERT Statoil Operational, history matching (Petroleum)
OpenDA TU Delft Operational, general
Verdandi INRIA Biophysical DA
PyOSSE Edinburgh, Reading Earth-observation DA
SANGOMA Conglomerate* Unify DA research
EMPIRE Reading (Met) Research (high-dim)
MIKE DHI Oceanographic. Commercial?
OAK Liège Oceaonagraphic
Siroco OMP Oceaonagraphic
FilterPy R. Labbe Engineering, general intro to Kalman filter
DASoftware Yue Li, Stanford Matlab, large-scale
Pomp U of Michigan R, general state-estimation
PyIT CIPR Real-world petroleum DA (?)
Datum Raanes Matlab, personal publications
EnKF-Matlab Sakov Matlab, personal publications and intro
EnKF-C Sakov C, light-weight EnKF, off-line
IEnKS code Bocquet Python, personal publications
pyda Hickman Python, personal publications

The EnKF-Matlab and IEnKS codes have been inspirational in the development of DAPPER.

*: AWI/Liege/CNRS/NERSC/Reading/Delft

Contributors

Patrick N. Raanes, Colin Grudzien, Maxime Tondeur, Remy Dubois

If you use this software in a publication, please cite as follows.

@misc{raanes2018dapper,
  author = {Patrick N. Raanes and others},
  title  = {nansencenter/DAPPER: Version 0.8},
  month  = December,
  year   = 2018,
  doi    = {10.5281/zenodo.2029296},
  url    = {https://doi.org/10.5281/zenodo.2029296}
}

Publication list

Powered by

Python Numpy Pandas Jupyter

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

DA-DAPPER-0.9.3.tar.gz (217.3 kB view hashes)

Uploaded Source

Built Distribution

DA_DAPPER-0.9.3-py3-none-any.whl (252.2 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page