pyproject2conda

A script to convert pyproject.toml dependencies to environemnt.yaml files.

Overview

The main goal of pyproject2conda is to provide a means to keep all basic dependency information, for both pip based and conda based environments, in pyproject.toml. I often use a mix of pip and conda when developing packages, and in my everyday workflow. Some packages just aren't available on both. If you use poetry, I'd highly recommend poetry2conda.

Features

• Simple comment based syntax to add information to dependencies when creating environment.yaml

Status

This package is actively used by the author, but is still very much a work in progress. Please feel free to create a pull request for wanted features and suggestions!

Quick start

Use one of the following to install pyproject2conda:

$ pip install pyproject2conda

or

$ conda install -c conda-forge pyproject2conda

If using pip, to install with rich and shellingham support, either install them your self, or use:

$ pip install pyproject2conda[all]

The conda-forge distribution of typer (which pyproject2conda uses) installs rich and shellingham by default.

Example usage

Basic usage

Consider the toml file test-pyproject.toml.

[project]
name = "hello"
requires-python = ">=3.8,<3.11"
dependencies = [
"athing", # p2c: -p # a comment
"bthing", # p2c: -s "bthing-conda"
"cthing; python_version < '3.10'", # p2c: -c conda-forge

]

# ...

Note the comment lines # p2c:.... These are special tokens that pyproject2conda will analyze. The basic options are:

usage: -c [-h] [-c CHANNEL] [-p] [-s] [packages ...]

Parser searches for comments '# p2c: [OPTIONS] CONDA-PACKAGES

positional arguments:
  packages

options:
  -h, --help            show this help message and exit
  -c CHANNEL, --channel CHANNEL
                        Channel to add to the pyproject requirement
  -p, --pip             If specified, install pyproject dependency with pip
  -s, --skip            If specified skip pyproject dependency on this line

So, if we run the following, we get:

$ pyproject2conda yaml -f tests/data/test-pyproject.toml
channels:
  - conda-forge
dependencies:
  - bthing-conda
  - conda-forge::cthing
  - pip
  - pip:
      - athing

Note that other comments can be mixed in.

By default, the python version is not included in the resulting conda output. To include the specification from pyproject.toml, use --python-include infer option:

$ pyproject2conda yaml -f tests/data/test-pyproject.toml --python-include infer
channels:
  - conda-forge
dependencies:
  - python>=3.8,<3.11
  - bthing-conda
  - conda-forge::cthing
  - pip
  - pip:
      - athing

Alternate syntax: using table instead of comments

While using comments to mark options has the convenience of placing the changes right next to the dependency, it can becore a bit cumbersome. If you feel this way, then you can use an alternative method to map pip dependencies to conda dependencies. For this, use the tool.pyproject2.conda.dependencies table. For example, we can do the same thing as above with:

# ...
[tool.pyproject2conda.dependencies]
athing = { pip = true }
bthing = { skip = true, packages = "bthing-conda" }
cthing = { channel = "conda-forge" }
pytest = { channel = "conda-forge" }
matplotlib = { skip = true, packages = [
  "additional-thing; python_version < '3.9'",
  "conda-matplotlib"
] }
build = { channel = "pip" }
# ...

Specify python version

To specify a specific value of python in the output, pass a value with:

$ pyproject2conda yaml -f tests/data/test-pyproject.toml --python-include \
    python=3.9
channels:
  - conda-forge
dependencies:
  - python=3.9
  - bthing-conda
  - conda-forge::cthing
  - pip
  - pip:
      - athing

Note that this is for including python in the resulting environment file.

You can also constrain packages by the python version using the standard pyproject.toml syntax "...; python_version < 'some-version-number'". For is parsed for both the pip packages and conda packages:

$ pyproject2conda yaml -f tests/data/test-pyproject.toml --python-version 3.10
channels:
  - conda-forge
dependencies:
  - bthing-conda
  - pip
  - pip:
      - athing

It is common to want to specify the python version and include it in the resulting environment file. You could, for example use:

$ pyproject2conda yaml -f tests/data/test-pyproject.toml --python-version 3.10 \
    --python-include python=3.10
channels:
  - conda-forge
dependencies:
  - python=3.10
  - bthing-conda
  - pip
  - pip:
      - athing

Because this is common, you can also just pass the option -p/--python:

$ pyproject2conda yaml -f tests/data/test-pyproject.toml --python 3.10
channels:
  - conda-forge
dependencies:
  - python=3.10
  - bthing-conda
  - pip
  - pip:
      - athing

Adding extra conda dependencies and pip requirements

You can also add additional conda and pip dependencies with the flags -d/--deps and -r/--reqs, respectively. Adding the last example:

$ pyproject2conda yaml -f tests/data/test-pyproject.toml -d dep -r req
channels:
  - conda-forge
dependencies:
  - bthing-conda
  - conda-forge::cthing
  - dep
  - pip
  - pip:
      - athing
      - req

These will also obey dependencies like dep:python_version<={version}. Pass the flags multiple times to pass multiple dependencies.

Command "aliases"

The name pyproject2conda can be a bit long to type. For this reason, the package also ships with the alias p2c, which has the exact same functionality. Additionally, the subcommands can be shortened to a unique match:

$ p2c y -f tests/data/test-pyproject.toml --python 3.10
channels:
  - conda-forge
dependencies:
  - python=3.10
  - bthing-conda
  - pip
  - pip:
      - athing

You can also call with python -m pyproject2conda.

Installing extras

Given the extra dependency:

# ...
[project.optional-dependencies]
test = [
"pandas",
"pytest", # p2c: -c conda-forge

]
dev-extras = [
# p2c: -s "additional-thing; python_version < '3.9'" # additional pkg
## p2c: -s "another-thing" # skipped because of ## before p2c.
"matplotlib", # p2c: -s conda-matplotlib

]
dev = ["hello[test]", "hello[dev-extras]"]
dist-pypi = [
# this is intended to be parsed with --no-base option
"setuptools",
"build", # p2c: -p

]

# ...

and running the following gives:

$ pyproject2conda yaml -f tests/data/test-pyproject.toml -e test
channels:
  - conda-forge
dependencies:
  - bthing-conda
  - conda-forge::cthing
  - conda-forge::pytest
  - pandas
  - pip
  - pip:
      - athing

pyproject2conda also works with self referenced dependencies:

$ pyproject2conda yaml -f tests/data/test-pyproject.toml -e dev
channels:
  - conda-forge
dependencies:
  - additional-thing
  - bthing-conda
  - conda-forge::cthing
  - conda-forge::pytest
  - conda-matplotlib
  - pandas
  - pip
  - pip:
      - athing

This also shows that p2c comments without dependencies are also parsed. To comment out such lines, make sure p2c is preceded by ##.

Header in output

By default, pyproject2conda includes a header in most output files to note that the files are auto generated. No header is included by default when writing to standard output. To override this behavior, pass --header/--noheader:

$ pyproject2conda yaml -f tests/data/test-pyproject.toml --header
#
# This file is autogenerated by pyproject2conda
# with the following command:
#
#     $ pyproject2conda yaml -f tests/data/test-pyproject.toml --header
#
# You should not manually edit this file.
# Instead edit the corresponding pyproject.toml file.
#
channels:
  - conda-forge
dependencies:
  - bthing-conda
  - conda-forge::cthing
  - pip
  - pip:
      - athing

Usage within python

pyproject2conda can also be used within python:

>>> from pyproject2conda.requirements import ParseDepends
>>> p = ParseDepends.from_path("./tests/data/test-pyproject.toml")

# Basic environment
>>> print(p.to_conda_yaml(python_include="infer").strip())
channels:
  - conda-forge
dependencies:
  - python>=3.8,<3.11
  - bthing-conda
  - conda-forge::cthing
  - pip
  - pip:
      - athing

# Environment with extras
>>> print(p.to_conda_yaml(extras="test").strip())
channels:
  - conda-forge
dependencies:
  - bthing-conda
  - conda-forge::cthing
  - conda-forge::pytest
  - pandas
  - pip
  - pip:
      - athing

Configuration

pyproject2conda can be configured with a [tool.pyproject2conda] section in pyproject.toml. To specify conda channels use:

# ...
[tool.pyproject2conda]
channels = ['conda-forge']
# these are the same as the default values of `p2c project`
template_python = "py{py}-{env}"
template = "{env}"
style = "yaml"
# options
python = ["3.10"]
# Note that this is relative to the location of pyproject.toml
user_config = "config/userconfig.toml"
default_envs = ["test", "dev", "dist-pypi"]

[tool.pyproject2conda.envs.base]
style = ["requirements"]
# Note that the default value for `extras` is the name of the environment.
# To have no extras, either pass
# extras = []
# or
#
extras = false

#
# A value of `extras = true` also implies using the environment name
# as the extras.
[tool.pyproject2conda.envs."test-extras"]
extras = ["test"]
style = ["yaml", "requirements"]

[[tool.pyproject2conda.overrides]]
envs = ['test-extras', "dist-pypi"]
base = false

[[tool.pyproject2conda.overrides]]
envs = ["test", "test-extras"]
python = ["3.10", "3.11"]

Note that specifying channels at the command line overrides tool.pyproject2conda.channels.

You can also specify environments without the base dependencies (those under project.dependencies) by passing the --no-base flag. This is useful for defining environments for build, etc, that do not require the package be installed. For example:

# ...
dist-pypi = [
# this is intended to be parsed with --no-base option
"setuptools",
"build", # p2c: -p

]

# ...

These can be accessed using either of the following:

$ pyproject2conda yaml -f tests/data/test-pyproject.toml -e dist-pypi --no-base
channels:
  - conda-forge
dependencies:
  - setuptools
  - pip
  - pip:
      - build

or

>>> from pyproject2conda.requirements import ParseDepends
>>> p = ParseDepends.from_path("./tests/data/test-pyproject.toml")

# Basic environment
>>> print(p.to_conda_yaml(extras='dist-pypi', include_base=False).strip())
channels:
  - conda-forge
dependencies:
  - setuptools
  - pip
  - pip:
      - build

Creating multiple environments from pyproject.toml

pyproject2conda provides a means to create all needed environment/requirement files in one go. We configure the environments using the pyproject.toml files in the [tool.pyproject2conda] section. For example, example the configuration:

# ...
[tool.pyproject2conda]
channels = ['conda-forge']
# these are the same as the default values of `p2c project`
template_python = "py{py}-{env}"
template = "{env}"
style = "yaml"
# options
python = ["3.10"]
# Note that this is relative to the location of pyproject.toml
user_config = "config/userconfig.toml"
default_envs = ["test", "dev", "dist-pypi"]

[tool.pyproject2conda.envs.base]
style = ["requirements"]
# Note that the default value for `extras` is the name of the environment.
# To have no extras, either pass
# extras = []
# or
#
extras = false

#
# A value of `extras = true` also implies using the environment name
# as the extras.
[tool.pyproject2conda.envs."test-extras"]
extras = ["test"]
style = ["yaml", "requirements"]

[[tool.pyproject2conda.overrides]]
envs = ['test-extras', "dist-pypi"]
base = false

[[tool.pyproject2conda.overrides]]
envs = ["test", "test-extras"]
python = ["3.10", "3.11"]

run through the command pyproject2conda project (or p2c project):

$ p2c project -f tests/data/test-pyproject.toml --dry
# --------------------
# Creating requirements base.txt
athing
bthing
cthing;python_version<"3.10"
# --------------------
# Creating yaml py310-test-extras.yaml
channels:
  - conda-forge
dependencies:
  - python=3.10
  - conda-forge::pytest
  - pandas
# --------------------
# Creating yaml py311-test-extras.yaml
channels:
  - conda-forge
dependencies:
  - python=3.11
  - conda-forge::pytest
  - pandas
# --------------------
# Creating requirements test-extras.txt
pandas
pytest
# --------------------
# Creating yaml py310-test.yaml
channels:
  - conda-forge
dependencies:
  - python=3.10
  - bthing-conda
  - conda-forge::pytest
  - pandas
  - pip
  - pip:
      - athing
# --------------------
# Creating yaml py311-test.yaml
channels:
  - conda-forge
dependencies:
  - python=3.11
  - bthing-conda
  - conda-forge::pytest

 ...

Note that here, we have used the --dry option to just print the output. In production, you'd omit this flag, and files according to --template and --template-python would be used.

The options under [tool.pyproject2conda] follow the command line options (replace - with _). To specify an environment, you can either use the [tool.pyproject.envs."environment-name"] method, or, if the environment is the same as the "extras" name, you can just specify it under tool.pyproject2conda.default_envs:

[tool.pyproject2conda]
# ...
default_envs = ["test"]

is equivalent to

[tool.pyproject2conda.envs.test]
extras = ["tests"]

To specify a conda environment (yaml) file, pass style = "yaml" (the default). To specify a requirements file, pass style = "requirements". You can specify both to make both.

Options in a given tool.pyproject2conda.envs."environemnt-name" section override those at the tool.pyproject2conda level. So, for example:

# ...
[tool.pyproject2conda.envs."test-extras"]
extras = ["test"]
style = ["yaml", "requirements"]

# ...

will override use the two styles instead of the default of yaml.

You can also override options for multiple environments using the [[tools.pyproject2conda.overrides]] list. Just specify the override option(s) and the environments to apply them to. For example, above we specify that the base option is False for envs test-extras and dist-pypi, and that the python version should be 3.10 and 3.11 for envs test and test-extras.

So in all, options are picked up, in order, from the environment definition, then the overrides list, and finally, from the default options.

You can also define "user defined" configurations. This can be done through the option --user-config. This allows you to define your own environments outside of the (most likely source controlled) pyproject.toml file. For example, we have the option user_config=config/userconfig.toml.

[tool.pyproject2conda.envs."user-dev"]
extras = ["dev", "dist-pypi"]
deps = ["extra-dep"]
reqs = ["extra-req"]
name = "hello"

Note that the full path of this file is note that the path of the user_conifg file is relative to thempyproject.toml file. So, if the pyproject.toml file is at a/path/pyproject.toml, the path of user configuration files will be a/path/config/userconfig.toml. We then can run the following:

$ p2c project -f tests/data/test-pyproject.toml --dry --envs user-dev
# --------------------
# Creating yaml py310-user-dev.yaml
name: hello
channels:
  - conda-forge
dependencies:
  - python=3.10
  - bthing-conda
  - conda-forge::pytest
  - conda-matplotlib
  - extra-dep
  - pandas
  - setuptools
  - pip
  - pip:
      - athing
      - build
      - extra-req

CLI options

See command line interface documentation for details on the commands and options.

Related work

The application pyproject2conda is used in the development of the following packages:

• cmomy
• thermoextrap
• tmmc-lnpy
• module-utilities
• analphipy
• pyproject2conda itself!

Documentation

See the documentation for a look at pyproject2conda in action.

License

This is free software. See LICENSE.

Contact

The author can be reached at wpk@nist.gov.

Credits

This package was created using Cookiecutter with the usnistgov/cookiecutter-nist-python template.

Changelog

Changelog for pyproject2conda

Unreleased

See the fragment files in changelog.d

v0.11.0 — 2023-11-28

Added

• Can now access "build-system.requires" as an extra. This can be useful for creating isolated environments to build a package.

Changed

• Can now specify pip as a conda dependency. This is needed for cases that there are no pip dependencies in the environment, but you want it there for installing local packages. This may be the case if using conda-lock on an environment. Note that, much like python is always first in the dependency list, pip is always last.

v0.10.0 — 2023-11-17

Added

• Can now specify conda changes using tool.pyproject2conda.dependencies table. This is an alternative to using # p2c: comments.
• Refactored code. Split parser to requirements and overrides. Also cleaned up the parsing logic to hopefully make future changes simpler.

v0.9.0 — 2023-11-14

Added

• Default is now to remove whitespace from dependencies. For example, the dependency module > 0.1 will become module>0.1. To override this behaviour, pass the option --no-remove-whitespace.
• Now supports python version >3.8,<=3.12
• Can now specify extras = false in pyprojec.toml to skip any extras. The default (extras = true) is the same as extras = [env_name] where env_name is the name of the environment (e.g., tool.pyproject2conda.envs.env_name).

v0.8.0 — 2023-10-02

Added

• Added option to either raise error, or print message for environments with no dependencies.

Changed

• pyproject2conda now works with pyproject.toml files with no dependencies.

v0.7.0 — 2023-09-26

Added

• Now use logging to print info output.

Changed

• cli now uses typer. Since the program was already typed, this simplifies the interface.
• Program can now be called with any of pyproject2conda, p2c, or python -m pyproject2conda.
• Added cli options to web documentation.
• Fixed small typos and typing issues.
• The cli option --python-include now requires an argument. This is due to typer not liking options with zero or one arguments. Instead of the bare flag --python-include including the python spec from pyproject.toml, you have to pass --python-include infer to get that behavior.
• Added extra all to pip install options. The default is to not include rich or shellingham. Using pip install pyproject2conda[all] includes these optional packages. Note that the conda-forge recipe is based on the plain install (i.e., no rich or shellingham). However, the conda-froge recipe for typer does include these. That means, if you want to install pyproject2conda without the optional extras, you'll have to use pip.

v0.6.1 — 2023-09-22

Changed

• Fixed edge case where --overwrite=check and have a user_config. Now when using p2c project with a user_config and overwrite=check, the timestamp of the output file will be compared to both the filename=pyproject.toml and user_config.

v0.6.0 — 2023-09-19

Added

• Added project subcommand. This uses a configuration in pyproject.toml to build multiple enivonments in one go.
• Added --deps and --reqs flags to include extra conda and pip requirements.
• Added --overwrite to check if output file exists.
• Now (correctly) using rich_click.
• Added tests for all new cases, and some edge cases.

v0.5.1 — 2023-09-09

Added

• Added --sort/--no-sort flag to cli. Default is to sort dependencies. This fixes issues with changing order in pyproject.toml leading to different yaml files.

Changed

• Changed structure of the repo to better support some third party tools.

• Moved nox environments from .nox to .nox/{project-name}/envs. This fixes issues with ipykernel giving odd names for locally installed environments.

• Moved repo specific dot files to the config directory (e.g., .noxconfig.toml to config/userconfig.toml). This cleans up the top level of the repo.

• added some support for using nbqa to run mypy/pyright on notebooks.

• Added ability to bootstrap development environment using pipx. This should simplify initial setup. See Contributing for more info.

• Main repo now on usnistgov.

This software was developed by employees of the National Institute of Standards and Technology (NIST), an agency of the Federal Government. Pursuant to title 17 United States Code Section 105, works of NIST employees are not subject to copyright protection in the United States and are considered to be in the public domain. Permission to freely use, copy, modify, and distribute this software and its documentation without fee is hereby granted, provided that this notice and disclaimer of warranty appears in all copies.

THE SOFTWARE IS PROVIDED 'AS IS' WITHOUT ANY WARRANTY OF ANY KIND, EITHER EXPRESSED, IMPLIED, OR STATUTORY, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY THAT THE SOFTWARE WILL CONFORM TO SPECIFICATIONS, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND FREEDOM FROM INFRINGEMENT, AND ANY WARRANTY THAT THE DOCUMENTATION WILL CONFORM TO THE SOFTWARE, OR ANY WARRANTY THAT THE SOFTWARE WILL BE ERROR FREE. IN NO EVENT SHALL NIST BE LIABLE FOR ANY DAMAGES, INCLUDING, BUT NOT LIMITED TO, DIRECT, INDIRECT, SPECIAL OR CONSEQUENTIAL DAMAGES, ARISING OUT OF, RESULTING FROM, OR IN ANY WAY CONNECTED WITH THIS SOFTWARE, WHETHER OR NOT BASED UPON WARRANTY, CONTRACT, TORT, OR OTHERWISE, WHETHER OR NOT INJURY WAS SUSTAINED BY PERSONS OR PROPERTY OR OTHERWISE, AND WHETHER OR NOT LOSS WAS SUSTAINED FROM, OR AROSE OUT OF THE RESULTS OF, OR USE OF, THE SOFTWARE OR SERVICES PROVIDED HEREUNDER.

Distributions of NIST software should also include copyright and licensing statements of any third-party software that are legally bundled with the code in compliance with the conditions of those licenses.