Tekton Compiler for Kubeflow Pipelines
Project description
Kubeflow Pipelines SDK for Tekton
The Kubeflow Pipelines SDK allows data scientists to define end-to-end machine learning and data pipelines. The output of the Kubeflow Pipelines SDK compiler is YAML for Argo.
The kfp-tekton SDK is extending the Compiler and the Client of the Kubeflow
Pipelines SDK to generate Tekton YAML
and to subsequently upload and run the pipeline with the Kubeflow Pipelines engine
backed by Tekton.
Table of Contents
- Extensions to the Kubeflow Pipelines SDK
- Project Prerequisites
- Installation
- Compiling a Kubeflow Pipelines DSL Script
- Running the Compiled Pipeline on a Tekton Cluster
- List of Available Features
- Tested Pipelines
- Troubleshooting
Extensions to the Kubeflow Pipelines SDK
In addition to the functionality provided by the Kubeflow Pipelines
SDK the kfp-tekton
SDK provides a TektonCompiler and a TektonClient:
TektonCompiler:
kfp_tekton.compiler.TektonCompiler.compilecompiles Python DSL code into a YAML file containing a TektonPipelineRunwhich can be deployed directly to a Tekton enabled Kubernetes cluster or uploaded to the Kubeflow Pipelines dashboard with the Tekton backend.
TektonClient:
kfp_tekton.TektonClient.create_run_from_pipeline_funccompiles DSL pipeline function and runs the pipeline on a Kubernetes cluster with KFP and Tekton
Project Prerequisites
- Python:
3.5.3or later - Tekton:
v0.14.0or later - Tekton CLI:
0.11.0 - Kubeflow Pipelines: KFP with Tekton backend
Follow the instructions for installing project prerequisites and take note of some important caveats.
Installation
You can install the latest release of the kfp-tekton compiler from
PyPi. We recommend to create a Python
virtual environment first:
python3 -m venv .venv
source .venv/bin/activate
pip install kfp-tekton
Alternatively you can install the latest version of the kfp-tekton compiler
from source by cloning the repository https://github.com/kubeflow/kfp-tekton:
-
Clone the
kfp-tektonrepo:git clone https://github.com/kubeflow/kfp-tekton.git cd kfp-tekton -
Setup Python environment with Conda or a Python virtual environment:
python3 -m venv .venv source .venv/bin/activate -
Build the compiler:
pip install -e sdk/python -
Run the compiler tests (optional):
pip install pytest make test
Compiling a Kubeflow Pipelines DSL Script
The kfp-tekton Python package comes with the dsl-compile-tekton command line
executable, which should be available in your terminal shell environment after
installing the kfp-tekton Python package.
If you cloned the kfp-tekton project, you can find example pipelines in the
samples folder or under sdk/python/tests/compiler/testdata folder.
dsl-compile-tekton \
--py sdk/python/tests/compiler/testdata/parallel_join.py \
--output pipeline.yaml
Note: If the KFP DSL script contains a __main__ method calling the
kfp_tekton.compiler.TektonCompiler.compile() function:
if __name__ == "__main__":
from kfp_tekton.compiler import TektonCompiler
TektonCompiler().compile(pipeline_func, "pipeline.yaml")
... then the pipeline can be compiled by running the DSL script with python3
executable from a command line shell, producing a Tekton YAML file pipeline.yaml
in the same directory:
python3 pipeline.py
Running the Compiled Pipeline on a Tekton Cluster
After compiling the sdk/python/tests/compiler/testdata/parallel_join.py DSL script
in the step above, we need to deploy the generated Tekton YAML to our Kubernetes
cluster with kubectl. The Tekton server will automatically start a pipeline run.
We can then follow the logs using the tkn CLI.
kubectl apply -f pipeline.yaml
tkn pipelinerun logs --last --follow
Once the Tekton Pipeline is running, the logs should start streaming:
Waiting for logs to be available...
[gcs-download : main] With which he yoketh your rebellious necks Razeth your cities and subverts your towns And in a moment makes them desolate
[gcs-download-2 : main] I find thou art no less than fame hath bruited And more than may be gatherd by thy shape Let my presumption not provoke thy wrath
[echo : main] Text 1: With which he yoketh your rebellious necks Razeth your cities and subverts your towns And in a moment makes them desolate
[echo : main]
[echo : main] Text 2: I find thou art no less than fame hath bruited And more than may be gatherd by thy shape Let my presumption not provoke thy wrath
[echo : main]
List of Available Features
To understand how each feature is implemented and its current status, please visit the FEATURES doc.
Tested Pipelines
We are testing the compiler on more than 80 pipelines
found in the Kubeflow Pipelines repository, specifically the pipelines in KFP compiler
testdata folder, the KFP core samples and the samples contributed by third parties.
A report card of Kubeflow Pipelines samples that are currently supported by the kfp-tekton
compiler can be found here.
If you work on a PR that enables another of the missing features please ensure that
your code changes are improving the number of successfully compiled KFP pipeline samples.
Troubleshooting
-
When you encounter ServiceAccount related permission issues, refer to the "Servince Account and RBAC" doc
-
If you run into the error
bad interpreter: No such file or directorwhen trying to use Python's venv, remove the current virtual environment in the.venvdirectory and create a new one usingvirtualenv .venv -
For big data passing, PVs must be created manually, or dynamic volume provisioning, has to be enabled (https://kubernetes.io/docs/concepts/storage/dynamic-provisioning). The PVC name has to match the PipelineRun name until issue #181 has been addressed.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
