Covalent Slurm Plugin
Project description
Covalent Slurm Plugin
Covalent is a Pythonic workflow tool used to execute tasks on advanced computing hardware. This executor plugin interfaces Covalent with HPC systems managed by Slurm. In order for workflows to be deployable, users must have SSH access to the Slurm login node, writable storage space on the remote filesystem, and permissions to submit jobs to Slurm.
To use this plugin with Covalent, simply install it using pip:
pip install covalent-slurm-plugin
The following shows an example of how a user might modify their Covalent configuration to support Slurm:
[executors.slurm]
username = "user"
address = "login.cluster.org"
ssh_key_file = "/home/user/.ssh/id_rsa"
remote_workdir = "/scratch/user"
cache_dir = "/tmp/covalent"
[executors.slurm.options]
partition = "general"
cpus-per-task = 4
gres = "gpu:v100:4"
exclusive = ""
parsable = ""
The first stanza describes default connection parameters for a user who is able to successfully connect to the Slurm login node using ssh -i /home/user/.ssh/id_rsa user@login.cluster.org. The second stanza describes default parameters which are used to construct a Slurm submit script. In this example, the submit script would contain the following preamble:
#!/bin/bash
#SBATCH --partition=general
#SBATCH --cpus-per-task=4
#SBATCH --gres=gpu:v100:4
#SBATCH --exclusive
#SBATCH --parsable
Within a workflow, users can then decorate electrons using these default settings:
import covalent as ct
@ct.electron(executor="slurm")
def my_task(x, y):
return x + y
or use a class object to customize behavior scoped to specific tasks:
executor = ct.executor.SlurmExecutor(
remote_workdir="/scratch/user/experiment1",
options={
"partition": "compute",
"cpus-per-task": 8
}
)
@ct.electron(executor=executor)
def my_custom_task(x, y):
return x + y
For more information about how to get started with Covalent, check out the project homepage and the official documentation.
Release Notes
Release notes are available in the Changelog.
Citation
Please use the following citation in any publications:
W. J. Cunningham, S. K. Radha, F. Hasan, J. Kanem, S. W. Neagle, and S. Sanand. Covalent. Zenodo, 2022. https://doi.org/10.5281/zenodo.5903364
License
Covalent is licensed under the GNU Affero GPL 3.0 License. Covalent may be distributed under other licenses upon request. See the LICENSE file or contact the support team for more details.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for covalent-slurm-plugin-0.8.0rc0.tar.gz
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 6b13862d87849b4ff5405451c6691bfa9fafe90cd3dc570d464eab8b8b9933c9 |
|
| MD5 | 7041a4d71970c050b71e7624e3f24814 |
|
| BLAKE2b-256 | 9332ec7683579c180a6f8a878ed6b819582ef40e2393adecc21e95677f293794 |
