Covalent ECS Plugin
Project description
Covalent ECS Plugin
Covalent is a Pythonic workflow tool used to execute tasks on advanced computing hardware. This executor plugin interfaces Covalent with AWS Elastic Container Service where the tasks are run using Fargate. In order for workflows to be deployable, users must have AWS credentials attached to the CovalentECSExecutorPolicy. Users will need additional permissions to provision or manage cloud infrastructure used by this plugin.
To use this plugin with Covalent, install it with pip:
pip install covalent-ecs-plugin
Users need to modify the entries to their Covalent configuration to support the ECS plugin. Below is an example which works using some basic infrastructure created for testing purposes:
[executors.ecs]
credentials = "/home/user/.aws/credentials"
profile = "default"
s3_bucket_name = "covalent-fargate-task-resources"
ecr_repo_name = "covalent-fargate-task-images"
ecs_cluster_name = "covalent-fargate-cluster"
ecs_task_family_name = "covalent-fargate-tasks"
ecs_task_execution_role_name = "ecsTaskExecutionRole"
ecs_task_role_name = "CovalentFargateTaskRole"
ecs_task_subnet_id = "subnet-871545e1"
ecs_task_security_group_id = "sg-0043541a"
ecs_task_log_group_name = "covalent-fargate-task-logs"
vcpu = 0.25
memory = 0.5
cache_dir = "/tmp/covalent"
poll_freq = 10
Within a workflow, users can then decorate electrons using these default settings:
import covalent as ct
@ct.electron(executor="ecs")
def my_task(x, y):
return x + y
or use a class object to customize the resources and other behavior:
executor = ct.executor.ECSExecutor(
vcpu=1,
memory=2,
ecs_task_subnet_id="subnet-871545e1",
ecs_task_security_group_id="sg-0043541a"
)
@ct.electron(executor=executor)
def my_custom_task(x, y):
return x + y
Ensure that Docker is running on the client side machine before deploying the workflow.
For more information about how to get started with Covalent, check out the project homepage and the official documentation.
Release Notes
Release notes are available in the Changelog.
Citation
Please use the following citation in any publications:
W. J. Cunningham, S. K. Radha, F. Hasan, J. Kanem, S. W. Neagle, and S. Sanand. Covalent. Zenodo, 2022. https://doi.org/10.5281/zenodo.5903364
License
Covalent is licensed under the GNU Affero GPL 3.0 License. Covalent may be distributed under other licenses upon request. See the LICENSE file or contact the support team for more details.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for covalent-ecs-plugin-0.12.0rc0.tar.gz
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | b5d1e87f7987be1b0ea4d11afda96e17cfc09b4fad1ad00252f21b03b30f941c |
|
| MD5 | 70d11fe771f9d536a04e1e163df6a924 |
|
| BLAKE2b-256 | 43291e4a02522b08c68fc9672241723478af650055f6c2584b4cd5a7d1078df0 |
