Scrapy extension Feed Exporter Storage Backend to export items to a Azure blob container
Project description
scrapy-feedexporter-azure is a Scrapy Feed Exporter Storage Backend that allows you to export Scrapy items to an Azure Blob Container.
Using scrapy-feedexporter-azure-blob
Add a the following to your Scrapy settings:
FEED_STORAGES = {"azure": "scrapy_feedexporter_azure_blob.AzureBlobFeedStorage"} FEED_URI = "azure://accountname/container" AZURE_ACCOUNT_NAME = "mycrawldata" # this is the subdomain to https://*.blob.core.windows.net/ AZURE_ACCOUNT_KEY = "your_account_key" AZURE_CONTAINER = 'sites' # the name of the container (you should have already created it) AZURE_FILENAME = 'bob.json' # the name of the file as it will be in blob storage
Note, you can define all of these settings when you run your crawler with the -s command line switch:
scrapy crawl <crawler_name> -s AZURE_ACCOUNT_KEY='<account_key>' -s AZURE_ACCOUNT_NAME='<account_name>'
TODO
Have the feed exporter properly parse the FEED_URI string: azure://account_name:api_key@container/filename.json
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for scrapy-feedexporter-azure-blob-0.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9700b6a794c9d190f04d3cea9d85b8c9e365bda92a08f230f36566f622411822 |
|
MD5 | 7f5a052f4cea67f825cb2eb4c323ce80 |
|
BLAKE2b-256 | 1166037f48a9e65a2fd6ff05cb585a5efe3aeeaed7f8ab03b0f0ce02c257a5d6 |
Close
Hashes for scrapy_feedexporter_azure_blob-0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 76ff25471922bb6ddbe34e7731a34f5a1fdbd3145dc2d11a72150f1c3c067dc4 |
|
MD5 | 18e9cfb793fcd496b8a0368f778f2f61 |
|
BLAKE2b-256 | 86a6771686b41aa0dbd9c5f110ce9e31d91044ced93c6f3c050bb6837f5fa61e |