log archiver for pypi cdn logs
Project description
# PyPI CDN Log Archiver
Get raw logs from S3 like object store, written by the cdn
Archive and compress them
Upload archive back to S3 like object store, in a sane structure
Remove raw logs that were archived
# Usage
You’ll need to export some env variables to get started:
$ export ACCESS_KEY=<your_access_key> $ export SECRET_KEY=<your_access_key> $ export PYPI_LOG_BUCKET=<bucket_name> $ export S3_HOST=<s3_like_host> (e.g. s3.amazonaws.com, objects.dreamhost.com)
Optionally you can set the date to archive like so:
$ export PYPI_LOG_DATE=2014-02-11
You can enable some debug output with:
$ export DEBUG=10
If you don’t set PYPI_LOG_DATE, the script will default to looking for yesterdays date.
Now you should be able to run it:
$ python pypi-cdn-log-archiver.py 112373 bytes written to testpypi-cdn-logs/archive/2014/2
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pypi-cdn-log-archiver-0.1.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | c636fc23d53f66324f1e2bc5b52e365a9e8608983ffd8510782ce0b1c5e9e995 |
|
MD5 | bfde3f6cb667bf3030122b11dcac8431 |
|
BLAKE2b-256 | d9ac066af7ce92d171cacdd318ea0cd70bce7c8bb138c0fc31a5f34c0b039471 |
Hashes for pypi_cdn_log_archiver-0.1.1-py2.7.egg
Algorithm | Hash digest | |
---|---|---|
SHA256 | 96f2273fedcf62d8fa8ccd210ab75baebbd9679fbc1d2053ddd977be041f8747 |
|
MD5 | 7f594b5a577c3245176b0b106f97b895 |
|
BLAKE2b-256 | e7d23e7ffa5686d4bf13b3b5af8e237a52ced324150bb6575cc7c41c96931809 |
Hashes for pypi-cdn-log-archiver-0.1.1.macosx-10.8-x86_64.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | e1dad40dc507a10c2136721665944fd0c28119bfa55dd68f06b9d13e4f0d01e1 |
|
MD5 | 4580bef75f783435418ce0c564cff23c |
|
BLAKE2b-256 | 6c91672e96f4b75fb6b0e9440ccc956474f30a707edeffe8e7cc64cd2af87179 |