Use a random User-Agent provided by fake-useragent every request
Project description
scrapy-fake-useragent
Random User-Agent middleware based on fake-useragent. It picks up User-Agent strings based on usage statistics from a real world database.
Configuration
Turn off the built-in UserAgentMiddleware and add RandomUserAgentMiddleware.
In Scrapy >=1.0:
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
}
In Scrapy <1.0:
DOWNLOADER_MIDDLEWARES = {
'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
}
Usage with scrapy-proxies
To use with middlewares of random proxy such as scrapy-proxies, you need:
set RANDOM_UA_PER_PROXY to True to allow switch per proxy
set priority of RandomUserAgentMiddleware to be greater than scrapy-proxies, so that proxy is set before handle UA
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for scrapy-fake-useragent-1.0.2.1.tar.gz
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | a8f6c6feeba04b4ea9c67509c8cdab836707b37dec7c90dfa30822e5a97a2318 |
|
| MD5 | 11c1e4d83f08bad13d189a62b2b1971c |
|
| BLAKE2b-256 | 7e08e87cbaa473ba2c9d13c1555d181f64657778a8aacd5036401b8ba1514614 |
Close
Hashes for scrapy_fake_useragent-1.0.2.1-py2.py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | f1a9b4cb51eb3337672b364fd5022ba0116d9f1e8631fa0935e69d9f70e86bc7 |
|
| MD5 | 065eed0a6df8106fcd029f1717270c66 |
|
| BLAKE2b-256 | 47619018a06f5e65019f473c2f9c07b98c516c849913253e9b5cc7549b8b3075 |
