Skip to main content

Parametrize and run scripts as notebooks with jupytext and papermill

Project description

Parametrize and run scripts as notebooks with jupytext and papermill

Build Status codecov.io Language grade: Python Pypi pyversions Binder

This is on-going research on how to run scripts as notebooks using Jupytext and Papermill.

The corresponding GitHub issues are Jupytext #231 and Papermill #365.

Note that the below is a Bash Jupyter notebook that is tested on our CI. If you wish, you can also open and run it interactively on Binder.

cd demo

Jupytext and Papermill

It is possible to convert a script to a notebook using Jupytext, and then to run it using Papermill. Here we use --set-kernel - to use the kernel that matches the current Python environment.

jupytext script.py -o notebook.ipynb --set-kernel -
papermill notebook.ipynb executed_notebook.ipynb -p integer 2 -p text 'updated text'
python -c "import nbformat
with open('executed_notebook.ipynb') as fp:
    nb = nbformat.read(fp, as_version=4)
print(nb.cells[-1]['outputs'][0]['text'])"

Inject parameters in a script using Papermill

Papermill needs the language information in the kernel to inject the parameters in the notebooks, so in this case we also need to set a kernel for the notebook.

jupytext script.py -o notebook.ipynb --set-kernel -
papermill notebook.ipynb notebook_with_parameters.ipynb --prepare-only -p integer 3 -p text 'updated text, v3'

Once the parameters have been injected, we can convert back the notebook to a script, and drop the kernel information:

jupytext notebook_with_parameters.ipynb -o script_with_parameters.py --update-metadata '{"kernelspec":null, "jupytext":null}'
cat script_with_parameters.py

And finally, we run the script using the Python interpreter:

python script_with_parameters.py

Towards Papermill + Jupytext?

With the papermill_jupytext package, we can open Jupytext scripts with the txt:// address. A kernel pointing to the current Python environment is injected in the documents that have no kernel.

papermill txt://script.py executed_notebook.ipynb  -p integer 4 -p text 'updated text, v4'
python -c "import nbformat
with open('executed_notebook.ipynb') as fp:
    nb = nbformat.read(fp, as_version=4)
print(nb.cells[-1]['outputs'][0]['text'])"

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

papermill_jupytext-0.0.1.tar.gz (3.7 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page