cratesmirror 1.1.0

CratesMirror

About

Download all crates on crates.io

Dependencies

Python >= 2.7.9

• requests

• GitPython

Installation

# pip install cratesmirror

Usage

from cratesmirror import CratesMirror

indexdir = '/srv/git/index'
cratesdir = '/srv/www/crates'
config = {'dl': 'https://crates.mirrors.ustc.edu.cn/api/v1/crates',
          'api': 'https://crates.io'}
# By default, it will be saved at os.getcwd()/crates.db
dbpath = '/var/lib/cratesmirror/crates.db'

with CratesMirror(indexdir, cratesdir, config=config, dbpath=dbpath) as mirror:
    mirror.update_repo()

# with proxy
proxies = {
  "http": "http://10.10.1.10:3128",
  "https": "http://10.10.1.10:1080",
}
with CratesMirror(indexdir, cratesdir, config=config, proxy=proxies, dbpath=dbpath) as mirror:
    mirror.update_repo()

Or use it in CLI

$ cratesmirror -h

usage: cratesmirror [-h] [-i INDEX] [-w CRATES] [-d DBPATH] [-f LOGFILE]
                    [-c CHECKDB] [-v]

optional arguments:
  -h, --help            show this help message and exit
  -i, --index INDEX     registry index directory (default: /srv/git/index)
  -w, --crates CRATES   crates directory (default: /srv/www/crates)
  -d, --dbpath DBPATH   database file path (default: None)
  -f, --logfile LOGFILE
                        log file path (default: None)
  -c, --checkdb CHECKDB
                        check database for missing crates (default: False)
  -v, --verbose

Available environment variables: HTTP_PROXY, HTTPS_PROXY, CRATES_DL, CRATES_API


Examples:
// Download all crates only
$ cratesmirror -d /var/lib/crates/crates.db -f /var/log/crates/debug.log

// Find out missing crates and update the repository
$ cratesmirror --checkdb -d /var/lib/crates/crates.db -f /var/log/crates/debug.log

// Update repo and commit custom settings
$ CRATES_DL='https://crates.mirrors.ustc.edu.cn/api/v1/crates' \
      cratesmirror -d /var/lib/crates/crates.db -f /var/log/crates/debug.log

Note

• By default, the script will:

  • assume that registry index directory is located at /srv/git/index, and crates are saved at /srv/www/crates

  • save downloaded crate as <CratesDir>/{name}/{name}-{version}.crate

  • save the database file at os.getcwd()/crates.db

• If the environment variable CRATES_DL or CRATES_API is set, its value will be saved at <IndexDir>/config.json and the changes will be committed automatically.

• After the first run, all you need to do is to run this script periodically using crontab-like tools or systemd.timers to sync with upstream.