A Python (3.7+ only) package for PETRA-III beamlines at DESY for automatically triggering online data processing jobs like remote data analysis, and local visualization tasks.
Project description
MENTO is a Python (3.7+ only) package for automatically triggering online processing jobs like
remote data analysis, and
local visualization tasks,
right from the beamline.
The package contains modules and utility functions that automate the boring bits like
getting access to remote HPC resources
submitting slurm jobs to run online analysis
The package is designed to work on all PETRA III beamlines at DESY.
Here is a short YouTube video about the main uses of MENTO, from a talk given at NOBUGS 2022.
Usage
In its simplest form, MENTO can be used at a PETRA-III beamline to launch a data processing job on a remote HPC (Maxwell) node, without needing to know how to connect to the HPC cluster Maxwell, or how to authenticate without having a scientific account for Maxwell, or indeed how job submissions work.
A remote analysis program can be started on data acquired at the beamline, by a Python script importing a MENTO module, creating a MENTO Trigger object and calling run() on it.
Here is a minimum working example:
from mento import Trigger my_trigger = Trigger() # default trigger will submit Slurm jobs on an HPC node my_analysis = 'maxwell_program.exe' # path to analysis program available on Maxwell my_args = ['analysis', 'parameters', 'list', 'inputdatafile'] # arguments required by analysis program, including input data my_trigger.run([my_analysis, my_args]) # connect to a remote machine with appropriate credentials and start the analysis job there
Note that paths to input data files and output directories can be specified as you see them at the beamline, and they will be automagically found in the remote core filesystem on Maxwell.
Paths to the analysis program on Maxwell, will of course need to refer to the ‘remote’ paths as seen from a Maxwell node, since MENTO cannot know about them from the beamline.
A dummy example of running MENTO to do live data analysis during data acquisition is shown in example.py.
Prerequisites for using MENTO for online data analysis
When using Maxwell to perform the data analysis, MENTO uses Maxwell’s Slurm system that manages access to the compute nodes and schedules processing jobs according to the permissions and resources granted to the user requesting the processing. Depending on the resources allocated to the user running MENTO, processing jobs may be queued on Maxwell, and run at a later time (or never).
Therefore, to have the data processing jobs run live at the same time as data acquisition during a beamtime, the recommendation is to request ‘online’ resources on Maxwell at the time of starting the beamtime. This is explained in detail in the DESY-IT/ASAP3 documentation for starting beamtimes here.
Here is the short version: when starting your beamtime, use the following command to request an ‘online’ Maxwell compute node which MENTO will then use to run remote data processing jobs:
startBeamtime --beamtimeId 999999999 --beamline P9999 --online
(Replace the beamline and beamtime ID with your beamline and beamtime ID, of course).
If your data processing program needs a GPU, please also add --feature gpu to the startBeamtime command.
To request ‘online’ Maxwell resources for commissioning runs instead of user beamtimes, please use startComissioning with the same arguments as for startBeamtime (--online and --feature).
Customizations
The MENTO trigger can be customized based on the beamline’s needs. An example:
Online results visualization
For online visualization of results, we need to automatically run plotting programs locally.
This can be quickly achieved by created a customized trigger using
my_trigger = Trigger(TriggerMethod.LOCAL_BASH_SCRIPT).
Calling run() on this trigger would then launch a visualization program (either directly or wrapped inside a Bash script) locally.
Of course, the input arguments and the parameter list would also need to be appropriately modified.
Installation
At PETRA III beamlines, the easiest way to install the MENTO package would be to use pip. The package can be downloaded as an artifact from the Gitlab repository page, or used directly in your pip command as follows:
python3 -m pip install desy-mento
API documentation
API documentation can be found in docs.
The documentation is auto-generated, so it should be as up-to-date as the comments in the source code. ;)
Contributing
Contributions in all forms are welcome - be it issues, bug reports, or code!
Citing
If you used MENTO and found it relevant to the work you are publishing, please consider citing the MENTO introduction article that appears in the SRI 2021 proceedings.
You can choose to either use the BibTeX citation given below, or use the CITATION.cff file to export the citation to your favourite format (BibTeX/APA/RIS/EndNote/CodeMeta).
@article{Vijay_Kartik_2022,
doi = {10.1088/1742-6596/2380/1/012104},
url = {https://dx.doi.org/10.1088/1742-6596/2380/1/012104},
year = {2022},
month = {dec},
publisher = {IOP Publishing},
volume = {2380},
number = {1},
pages = {012104},
author = {S Vijay Kartik and Michael Sprung and Fabian Westermeier and Anton Barty},
title = {MENTO: Automated near real-time data analysis at PETRA III},
journal = {Journal of Physics: Conference Series},
}
Licences
All code is licensed under GPL-3.0-or-later
All documentation is licensed under CC-BY-4.0
All other files are licensed under CC0-1.0
Full texts of the licences can be found in LICENSES.
This project aims to be REUSE compliant.
Contact
For questions and critiques, please contact S. Vijay Kartik <vijay.kartik@desy.de>.
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for desy_mento-0.2.1-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | e3139c7137182d1e603e35954005dc44f7d2c6b5855f514be00ecf920e863669 |
|
| MD5 | 7c55973a3fa1345e133e4cc689472627 |
|
| BLAKE2b-256 | 22df1a1cb48e05162f1c4862a642e57d329c0c1bf36aec92027309166a5693b4 |
