Dask plugin for flytekit
Project description
Flytekit Dask Plugin
Flyte can execute dask
jobs natively on a Kubernetes Cluster, which manages the virtual dask
cluster's lifecycle
(spin-up and tear down). It leverages the open-source Kubernetes Dask Operator and can be enabled without signing up
for any service. This is like running a transient (ephemeral) dask
cluster - a type of cluster spun up for a specific
task and torn down after completion. This helps in making sure that the Python environment is the same on the job-runner
(driver), scheduler and the workers.
To install the plugin, run the following command:
pip install flytekitplugins-dask
To configure Dask in the Flyte deployment's backed, follow step 1 and step 2
An example can be found in the documentation.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for flytekitplugins-dask-1.8.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2972dfec98bf8f68f142d037dfb64a65cefff01fc7852c88d2a27e59d28abaf2 |
|
MD5 | f53aca80208982732e47dae88c717171 |
|
BLAKE2b-256 | 3d004c8313bfe88c4a41937f86391cd17ccb51ee1ee3db90cf0cb5483f0c3146 |
Hashes for flytekitplugins_dask-1.8.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 83a47b4bd874fdbb968ce56e5d7956e17175f9287b4360bb2cedfa8a2a272cc6 |
|
MD5 | 1e50ee5c8b6044cc17d77cb9cc4469c2 |
|
BLAKE2b-256 | 19ab8adf451aba2a897eb55422daae0f8984582e73e6d46e3302c00b61689540 |