Scrapy extension Feed Exporter Storage Backend to export items to a Azure blob container
Project description
scrapy-feedexporter-azure is a Scrapy Feed Exporter Storage Backend that allows you to export Scrapy items to an Azure Blob Container.
Using scrapy-feedexporter-azure-blob
Add a the following to your Scrapy settings:
FEED_STORAGES = {"azure": "scrapy_feedexporter_azure_blob.AzureBlobFeedStorage"} FEED_URI = "azure://accountname/container" AZURE_ACCOUNT_NAME = "mycrawldata" # this is the subdomain to https://*.blob.core.windows.net/ AZURE_ACCOUNT_KEY = "your_account_key" AZURE_CONTAINER = 'sites' # the name of the container (you should have already created it) AZURE_FILENAME = 'bob.json' # the name of the file as it will be in blob storage
Note, you can define all of these settings when you run your crawler with the -s command line switch:
scrapy crawl <crawler_name> -s AZURE_ACCOUNT_KEY='<account_key>' -s AZURE_ACCOUNT_NAME='<account_name>'
TODO
Have the feed exporter properly parse the FEED_URI string: azure://account_name:api_key@container/filename.json
DEVELOPER NOTES
When deploying to pypi, just use twine
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for scrapy-feedexporter-azure-blob-0.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0bdabad498f63fd30e4776a48d3402674b50dcc976993e33241ccfa385b8b563 |
|
MD5 | bff68bd95f6bd6b6707c88694904af54 |
|
BLAKE2b-256 | 04af3004d2dc95446481451885a33e82fffc4904f51614f855d66094df5e5b5a |
Close
Hashes for scrapy_feedexporter_azure_blob-0.2-py2-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0ba79543654d6aa7bc932bf8649f209e8ef446e398c03465f1c7bc1be1293f1e |
|
MD5 | a89f97e013a12f95c69deaef82d64fd2 |
|
BLAKE2b-256 | 6c1a2d1f4144ecf7a315d5275dc09f3d1e3f13e462737db8639edf63c303821b |