Python functions and CLI to mirror git repositories available on HTTP(S) to S3
Project description
mirror-git-to-s3
Python functions and CLI to mirror public git repositories available on HTTP(S) to S3. Essentially converts smart protocol git repositories to the so-called dumb protocol. Does not use temporary disk space, and uses streaming under the hood. This should allow the mirroring to be run on systems that don't have much disk or available memory, even on repositories with large objects.
This project mirrors objects stored on Large File Storage (LFS). Note however LFS objects are not accessible via the dumb protocol. To work around this, you can use git-lfs-http-mirror that fires up a temporary local LFS server during git commands.
At the time of writing repositories with many objects can be slow to mirror.
Installation
pip install mirror-git-to-s3
Usage
To mirror one or more repositories from Python, use the mirror_repos
function, passing it an iterable of (source, target) mappings.
from mirror_git_to_s3 import mirror_repos
mirror_repos((
('https://example.test/my-first-repo', 's3://my-bucket/my-first-repo'),
('https://example.test/my-second-repo', 's3://my-bucket/my-second-repo'),
))
Once a repository is mirrored to a bucket that doesn't need authentication to read, it can be cloned using standard git commands using the virtual host or path of the S3 bucket.
git clone https://my-bucket.s3.eu-west-2.amazonaws.com/my-first-repo
Under the hood, boto3 is used to communicate with S3. The boto3 client is constructed automatically, but you can override the default by using the get_s3_client
argument.
import boto3
from mirror_git_to_s3 import mirror_repos
mirror_repos(mappings(), get_s3_client=lambda: boto3.client('s3'))
This can be used to mirror to S3-compatible storage.
import boto3
from mirror_git_to_s3 import mirror_repos
mirror_repos(mappings(), get_s3_client=lambda: boto3.client('s3', endpoint_url='http://my-host.com/'))
To mirror repositories from the the command line pairs of --source
--target
options can be passed to mirror-git-to-s3
.
mirror-git-to-s3 \
--source 'https://example.test/my-first-repo' --target 's3://my-bucket/my-first-repo' \
--source 'https://example.test/my-second-repo' --target 's3://my-bucket/my-second-repo'
At the time of writing, there is no known standard way of discovering a set of associated git repositories, hence to remain general, this project must be told the source and target addresses of each repository explicitly.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for mirror_git_to_s3-0.0.9-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 29d543c2866565060c76e3dc137705e0cf4371642cd2f8fe0acc75acc303c897 |
|
MD5 | 864b030948a4a38be0a803394ef55bfb |
|
BLAKE2b-256 | 411fe8d2e4f86bf35819dd83d5972e12e9e72f11774c318c0c86ea61bf04bcdf |