Skip to main content

Autosubmit: a versatile tool to manage Weather and Climate Experiments in diverse Supercomputing Environments

Project description

Autosubmit is a tool to create, manage and monitor experiments using
computing clusters, HPC platforms and supercomputers remotely via ssh.

Get involved or contact us:

Autosubmit GitLab: https://earth.bsc.es/gitlab/es/autosubmit
Autosubmit Mailing List: autosubmit@bsc.es

How to cite Autosubmit:

D. Manubens-Gil, J. Vegas-Regidor, C. Prodhomme, O. Mula-Valls and F. J. Doblas-Reyes,
"Seamless management of ensemble climate prediction experiments on HPC platforms,"
2016 International Conference on High Performance Computing & Simulation (HPCS),
Innsbruck, 2016, pp. 895-900. doi: 10.1109/HPCSim.2016.7568429

HOW TO DEPLOY/SETUP AUTOSUBMIT FRAMEWORK
========================================

- Autosubmit has been tested:

with the following Operating Systems:
* Linux Debian
* Linux OpenSUSE

on the following HPC platforms:
* Ithaca (IC3 machine)
* Stargate (IC3 machine)
* MareNostrum (BSC machine)
* MareNostrum3 (BSC machine)
* HECToR (EPCC machine)
* Lindgren (PDC machine)
* C2A (ECMWF machine)
* CCA (ECMWF machine)
* ARCHER (EPCC machine)

- Pre-requisites: These packages (bash, python2, sqlite3, git-scm > 1.8.2, subversion, dialog*) must be available at local
machine. These packages (argparse, dateutil, pyparsing, numpy, pydotplus, matplotlib, paramiko, saga-python,
python2-pythondialog*, mock, portalocker) must be available for python runtime. And the machine is also able to access
HPC platforms via password-less ssh.

*: optional

- Install Autosubmit
> pip install autosubmit
or download, unpack and "python setup.py install"

- Create a repository for experiments: Say for example "/cfu/autosubmit" then "autosubmit configure" and follow
instructions

- Create a blank database: "autosubmit install"

HOW TO USE AUTOSUBMIT
=====================
> autosubmit -h

> autosubmit expid --HPC ithaca --description "experiment is about..."

Say for example, "cxxx" is 4 character based expid generated by system automatically.
First character is a letter, the other three alfa-numeric characters allow to identify uniquely the experiment.

> vi /cfu/autosubmit/cxxx/conf/expdef_cxxx.conf

> vi /cfu/autosubmit/cxxx/conf/autosubmit_cxxx.conf

> vi /cfu/autosubmit/cxxx/conf/platforms_cxxx.conf

> vi /cfu/autosubmit/cxxx/conf/jobs_cxxx.conf

> autosubmit create cxxx

> nohup autosubmit run cxxx &

Cautions:
- Before launching autosubmit check the following stuff:
> ssh ithaca # (for example) check other HPC platforms where password-less ssh is feasible.
- After launching autosubmit, one must be aware of login expiry limit and policy (if applicable for any HPC)
and renew the login access accordingly (by using token/key etc) before expiry.

HOW TO MONITOR EXPERIMENT
=========================

> autosubmit monitor -h

> autosubmit monitor cxxx
or
> autosubmit monitor cxxx -o png

Above generated plot with date & time stamp can be found at:

/cfu/autosubmit/cxxx/plot/cxxx_date_time.pdf
or
/cfu/autosubmit/cxxx/plot/cxxx_date_time.png


HOW TO RESTART EXPERIMENT
=========================

> autosubmit recovery -h

> autosubmit recovery cxxx -s # saving the pickle file

> nohup autosubmit run cxxx &


FULL DOCUMENTATION AND HOW TOs
==============================

Check the Autosubmit documentation provided in the docs/ folder of the package, in PDF format.
Check the online documentation in the following web page: http://www.bsc.es/projects/earthscience/autosubmit/

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

autosubmit-3.7.7.tar.gz (1.6 MB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page