Skip to main content

CLI tool to upload Python packages listed in a requirements.txt file into a Sonatype Nexus (from Pypi), and return the list of the artifact URLs

Project description

image0 image1

Python tools to help with the development & deployment of company-private Python packages.

It was developped to use Sonatype Nexus as a private Pypi mirror (until it supports this natively), but should be adaptable to any repository supporting artifact upload with HTTP.

It is composed of 2 distincts :

  • nexus_uploader : a simple module to be used as a setup_requires entry in setup.py, in order to easily upload Python packages onto a Nexus.

  • pyRequirements2nexus : a CLI tool to convert standard Python requirements.txt into nexus-requirements.txt files, made of URLs pointing to installable packages that pyRequirements2nexus mirrored on your Nexus.

Features

nexus_uploader features

  • easy integration in setup.py

  • HTTP BasicAuth to connect to Sonatype Nexus

pyRequirements2nexus features

  • full dependency resolution of Python packages, working with both Pypi public ones & private ones on a Nexus

  • to install packages, the end machine only require a connexion to the Nexus host, not to the Internet

  • support -e editable packages in requirements.txt

  • support package URL fallbacks as comments in requirements.txt

  • a list of packages already included in the base environment can be specified (e.g. if you are using Anaconda), so that they will be excluded from the final nexus-requirements.txt

Limitations

  • only support == version locking in requirements.txt (not >=). This is intentional, to ensure package versions do not change unexpectedly.

How to upload home-made Python packages to my Nexus ?

Here is a handy recipe you can put in your setup.py :

setup(
    ...
    setup_requires=['nexus_uploader']
)

Usage:

$ python setup.py sdist nexus_upload --repository http://nexus/content/repositories/Snapshots/com/vsct/my_project --username $REPOSITORY_USER --password $REPOSITORY_PASSWORD

Important note

Contrary to requirements.txt files, there is no way to include URLs to dependencies in setup.py install_requires (nor setup_requires). To work around this limitation, pyrequirements2nexus takes advantage of the deprecated “dependency_links” mechanism. Using them, it is able to discover the full chain of Python packages dependencies, both from Pypi AND from your local Nexus.

To easily generate the “dependency_links.txt” in your Nexus-hosted private package out of your requirements.txt, use the following recipe in your its setup.py :

with open(os.path.join(os.path.dirname(os.path.realpath(__file__)), 'requirements.txt')) as requirements_file:
    requirements = requirements_file.readlines()
    dependency_links = [req for req in requirements if req.startswith('http')]
    install_requires = [req for req in requirements if not req.startswith('http')]

setup(
    ...
    install_requires=install_requires,
    dependency_links=dependency_links,
)

Because “dependency_links” are not supported since pip 1.6, they will NOT be installed by pip install normal recursive dependency-retrieval algorithm. You should always use the flat nexus-requirements.txt with pip install.

pyRequirements2nexus usage

pip install nexus_uploader
pyRequirements2nexus --help

Also take a look at jenkins-install-python-requirements.sh for an example of how we use it on our Jenkins.

Installation of nexus-requirements.txt on an end machine

pip install --user --no-index --no-deps --no-cache-dir --upgrade --requirement nexus-requirements.txt

Supported requirements.txt format

http://nexus/content/repositories/repo_id/my/project/group/mypkgname/1.0/mypkgname-1.0-py2.py3-none-any.tar.gz
nose==1.3.7   # -> transformed into an URL like this: http://nexus/content/repositories/repo_id/my/project/group/...
-e ../my/relative/path  # http://nexus/content/repositories/...fallback_url...

Contributing

pre-commit hooks installation:

pip install -r dev-requirements
pre-commit install

Unit tests:

PYTHONPATH=. py.test tests/

Smoke tests using Pypi:

PYTHONPATH=. ipython3 --pdb tests/smoke_test_extract_classifier_and_extension.py 200

FAQ

pip install - Download error on https://pypi.python.org / Couldn’t find index page for

The stack trace :

Collecting http://nexus/content/repositories/pip/com/vsct/pip/jsonschema/2.5.1/jsonschema-2.5.1-py2.py3-none-any.tar.gz (from -r scripts/requirements.pip (line 12))
  Downloading http://nexus/content/repositories/pip/com/vsct/pip/jsonschema/2.5.1/jsonschema-2.5.1-py2.py3-none-any.tar.gz (50kB)
    Complete output from command python setup.py egg_info:
    Download error on https://pypi.python.org/simple/vcversioner/: [Errno -2] Name or service not known -- Some packages may not be found!
    Couldn't find index page for 'vcversioner' (maybe misspelled?)
    Download error on https://pypi.python.org/simple/: [Errno -2] Name or service not known -- Some packages may not be found!
    No local packages or download links found for vcversioner

Explanation : https://github.com/Julian/jsonschema/issues/276

Solution :

$ cat <<EOF > ~/.pydistutils.cfg
[easy_install]
allow_hosts = nexus
find_links = http://nexus/content/repositories/pip/com/vsct/pip/vcversioner/2.14.0.0/
EOF

How to generate a “–default-packages” file out of an Anaconda .sh installer

grep -aF 'extract_dist ' Anaconda3-2.4.1-Linux-x86_64.sh \
    | perl -p -e 's/extract_dist (.+?[0-9])[^.]*$/\1\n/;' -e 's/^(.+)-(.+)$/\1 == \2/;' \
    | grep -vE '^(_cache|_license|anaconda|python) ' > anaconda3-2.4.1_included_packages.txt

Tip for easily removing packages from your nexus

pip install --user repositorytools
export REPOSITORY_USER=...
export REPOSITORY_PASSWORD=
artifact delete http://nexus/content/repositories/pip/com/vsct/pip/ultrajson/1.35/ultrajson-1.35-macosx-10.6-intel.tar.gz

ToDo

  • Python 2 support

  • detect if package releases on Pypi require gcc compilation (are they using setuptools/distutils Extension in setup.py ?)

  • classifier-based selection of Python packages

  • add support for md5 & sha1 upload/checks

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

nexus_uploader-1.0.8.tar.gz (14.9 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page