Skip to main content

Scrapy http proxy middleware that gets proxy parameters from settings

Project description

PyPI Version

Scrapy middleware that reads proxy config from settings.

Install:

pip install proxy-middleware

Usage: add it to DOWNLOADER_MIDDLEWARES in scrapy settings:

DOWNLOADER_MIDDLEWARES = {
   'proxy_middleware.ProxyFromSettingsMiddleware': 10,
   ...

Pass proxy config via HTTP_PROXY and HTTPS_PROXY settings variables. HTTPPROXY_AUTH_ENCODING is also respected:

scrapy crawl my-spider -s HTTP_PROXY=http://localhost:8118

License is MIT.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

proxy-middleware-0.1.1.tar.gz (1.8 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page