os-scrapy-aio-resolver

This project provide a async DNS resolver.

Install

pip install os-scrapy-aio-resolver

You can run example spider directly in the project root path

scrapy crawl example

Usage

Settings

• enable DNS resolver in the project settings file:

  DNS_RESOLVER = "os_scrapy_aio_resolver.AsyncResolver"
  

• use asyncio reactor

  TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
  

  or you can use os-scrapy command line to start crawling with -r asyncio

  os-scrapy crawl example -r asyncio
  

• enable/disable DNS cache, set cache size

  DNSCACHE_ENABLED = True
  DNSCACHE_SIZE = 10000
  

Unit Tests

sh scripts/test.sh

License

MIT licensed.