ReProx: ReProcessing for XENONnT

Package	CI

Documentation

Please visit the documentation for installation instructions and examples.

Examples

Can be found either on github or the online documentation.

Reprocessing on dali

Process data in so far available on dali with the current container

Logic

There are several (sequential) steps with (associated scripts):

• Step 1. Find runs to process (reprox-find-data)
• Step 2. Process the runs that were found (reprox-start-jobs)
• Step 3. Move the data that was processed to the desired folder (reprox-move-folders)

One can also run these three steps from one file (reprox-reprocess), which runs all three in order.

The best place to start is by going over these files and do reprox-find-data --help to see which options there are. Most are discussed below.

Running step by step

Below, we show how these three steps are done. This can also be done in one command skip to single command.

Step 0 - Activation and test installation

You only have to do it once, to prevent confusion we will go over it step by step.

First, activate a container (NB! the singularity containers do not work as they cannot communicate with the job submission of dali).

source /cvmfs/xenon.opensciencegrid.org/releases/nT/development/setup.sh
git clone git@github.com:XENONnT/reprox.git
pip install -e reprox --user

test that the installation is complete and successful

reprox-find-data --help

Trouble-shooting

Now, the commands above may sometimes not work as expected due to permission errors on the containers. If there is an error, you could see reprox-find-data: command not found. If this is the case, simply navigate to the bin folder of reprox and run the commands as below:

cd repox/bin
python reprox-find-data --help

The other reprox scripts are similarly located in the bin folder. If you had to change this once, you have to do python <script> for all the scripts listed below.

Step 1 - finding data to (re)process on dali

Now we have to know which data to process, this can be done with the following command. Determine which data to process:

reprox-find-data \
    --package cutax \
    --context xenonnt_v6 \ 
    --target event_info event_pattern_fit cuts_basic \
    --cmt-version global_v6

The --package and --context arguments specify where to load the context from (straxen/cutax) and which context to use. In this example, we use xenonnt_v6. The --target argument specifies which datatypes to produce. This can be a list as in the example above. We will check if the datatypes can be produced for this given context. Since some context may use a global CMT version that is only valid for a range of runs, the --cmt-version is specified separately and tells the script to only process runs that are valid in this cmt_version. This can be disabled using --cmt-version False (for example, you know that the CMT version is always valid for the datatypes you requested).

This takes a while (+/- 30 minutes) and writes a file called /dali/lgrandi/xenonnt/data_management_reprocessing/to_do_runs.csv (depending on your ini file). This file has a list of runs that you can process given the options as above.

Step 2 - starting the jobs to process the data

After producing /dali/lgrandi/xenonnt/data_management_reprocessing/to_do_runs.csv, we need to submit the jobs to process the data. Most of the arguments are the same as above, we now also specify some self-explanatory arguments for the jobs to be submitted.

reprox-start-jobs \
    --package cutax \
    --context xenonnt_v6 \
    --target event_info event_pattern_fit cuts_basic \
    --ram 12000 \
    --cpu 2

Step 3 - move to the production folder

Now, hopefully most of the data has been processed successfully, we can now move it to the production folder. This includes a check to see if the data was processed successfully so even if a few jobs failed (or are still running), you can safely run this command below.

reprox-move-folders

Run entire workflow (steps 1-3 in a single command)

You can also do all the above in a single command, using the same arguments (see above for explanation of each.).

reprox-reprocess \
    --package cutax \
    --context xenonnt_v6 \
    --target event_info event_pattern_fit cuts_basic \
    --cmt-version global_v6 \
    --ram 12000 \
    --cpu 2 \
    --move-after-workflow # To move the data into the production folder

Advanced usage

Below are several more advanced use cases.

Changing the defaults of processing

You might want to play with the config file that says how many resources one uses by default. The reprocessing.ini file. You can either change the source code of this file, or you can overwrite it as follows:

git clone git@github.com:XENONnT/reprox.git
cp reprox/reprox/reprocessing.ini my_reprocessing_config.ini

# # Edit my_reprocessing_config.ini. For example using vim:
# vi my_reprocessing_config.ini 

# overwrite the file used using an environment variable
export REPROX_CONFIG=$(pwd)/my_reprocessing_config.ini

You will see that your defaults have been changed (e.g. do reprox-reprocess --help) reflecting the changes you made in the .ini file.

Use custom config

You might want to process some data with slightly different settings, this can be done using the--context_kwargs argument as follows (please don't move it into the production folder unless you know what you are doing):

reprox-reprocess \
    --package cutax \
    --context xenonnt_v6 \
    --target event_info event_pattern_fit cuts_basic \
    --cmt-version global_v6 \
    --ram 12000 \
    --cpu 2 \ 
    --context-kwargs '{"s1_min_coincidence": 2, "s2_min_pmts": 10}'

Using reprox from your jupyter notebook

You can also run the commands from above in a notebook or python script.

from reprox import find_data, submit_jobs, validate_run

targets = 'event_info event_pattern_fit cuts_basic'.split()

# First determine which data to process
find_data.find_data(
    targets=targets,
    exclude_from_invalid_cmt_version='global_v6'
)
# Now start running the jobs
submit_jobs.submit_jobs(targets=targets)

# Finally move the jobs to the production folder
validate_run.move_all()

Processing NV data

By default, the package assumes that only linked-mode or TPC runs are processed, if you want to instead process NV data you need to tell the scripts to also take into account the NV detector:

reprox-reprocess \
    --package cutax \
    --context xenonnt_v6 \
    --target events_nv \
    --detectors neutron_veto muon_veto
    --ram 12000 \
    --cpu 2 \ 
    --cmt-version False

Using tagged versions

One might want to run with a different tag as so

MY_TAG=2021.12.2
source /cvmfs/xenon.opensciencegrid.org/releases/nT/$MY_TAG/setup.sh
reprox-reprocess \
    --package cutax  \
    --context xenonnt_v5  \
    --targets event_info \
    --cmt-version global_v5 \
    --ram 24000  \
    --cpu 2  \
    --move-after-workflow \
    --tag $MY_TAG

0.2.1 / 2022-03-29

• Fix kwargs setting (#53)

0.2.0 / 2022-03-29

• Fix deprecation warning (#50)
• Add timeout to jobs (#51)
• print reprox version (#52)
• Fix permissions (#49)
• Update coverage (#47)
• Run straxen version on coveralls (#48)
• Versions (#44, #45, #33, #43)

0.1.0 / 2022-01-27

• Finetuning after testing (#18)
• requirement updates (#22)

0.0.3 / 2022-01-17

• Bugfix data finding (#17)

0.0.2 / 2022-01-17

• Bugfix for kwargs in job submssion (#15
• Update documentation (#13, fb0cbdab46f92d87871baae298efa20e45168e6d, 55144dc8c00094e9f64af0bc45f85bebe2b3b1e5,adf34cc4a494ad761a709edb1a83597a594ca238, 5e492ea8a9d1db3e426210b90c2eb2147eb214d9)
• Refactored by Sourcery (#14)
• Update with py3.10 (#16)
• Release (#12)

0.0.1 / 2022-01-17

• Update documentation (#7, #11, 2895ebb5063f7951550c8e57059fdd3bbf600b9e)
• Add tests (#1, #9, #10)
• Pin requirements (#3, #4, #5, #6, #8)
• Release (#12)

0.0.0 / 2022-01-17

• Initial commits