Dask plugin for flytekit
Project description
Flytekit Dask Plugin
Flyte can execute dask
jobs natively on a Kubernetes Cluster, which manages the virtual dask
cluster's lifecycle
(spin-up and tear down). It leverages the open-source Kubernetes Dask Operator and can be enabled without signing up
for any service. This is like running a transient (ephemeral) dask
cluster - a type of cluster spun up for a specific
task and torn down after completion. This helps in making sure that the Python environment is the same on the job-runner
(driver), scheduler and the workers.
To install the plugin, run the following command:
pip install flytekitplugins-dask
To configure Dask in the Flyte deployment's backed, follow step 1 and step 2
An example can be found in the documentation.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for flytekitplugins-dask-1.4.0b1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2fdb307b205ed9ca50dffc8b50b8bad106e0e85db64b19f058da9e39c0dae7fb |
|
MD5 | 1f1ad4082569dce372fe9a754bb3aca6 |
|
BLAKE2b-256 | 106f78aab8f6ff08fc502aa75746fbe0077c5c1dffee7a08ab6320331abf855c |
Hashes for flytekitplugins_dask-1.4.0b1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bb3cddb980ece7fb571725cffbf8b88885acb2fe166bd89d68b85c14e2e2109c |
|
MD5 | b28bec6ec6e33344a78231a27ce9b415 |
|
BLAKE2b-256 | 9cc922a6d13bc3c3fc981e38fb089567438e89360b9d1db4e42d52db282e7372 |