skip to navigation
skip to content

scrapy-dotpersistence 0.3.0

Scrapy extension to sync `.scrapy` folder to an S3 bucket

Scrapy extension to sync .scrapy folder to an S3 bucket.

Installation

You can install scrapy-dotpersistence using pip:

pip install scrapy-dotpersistence

You can then enable the extension in your settings.py:

EXTENSIONS = {
    ...
    'scrapy_dotpersistence.DotScrapyPersistence': 0
}

How to use it

Enable extension through settings.py:

DOTSCRAPY_ENABLED = True

Configure the exension through settings.py:

ADDONS_AWS_ACCESS_KEY_ID = "ABC"
ADDONS_AWS_SECRET_ACCESS_KEY = "DEF"
ADDONS_AWS_USERNAME = "username"
ADDONS_S3_BUCKET = "test-bucket-name"

You can change a dotpersistence folder path with environ:

export DOTSCRAPY_DIR='/tmp/.scrapy'
 
File Type Py Version Uploaded on Size
scrapy-dotpersistence-0.3.0.tar.gz (md5) Source 2016-08-04 2KB
scrapy_dotpersistence-0.3.0-py2-none-any.whl (md5) Python Wheel py2 2016-08-04 4KB