skip to navigation
skip to content

reppy 0.4.7

Replacement robots.txt Parser

Replaces the built-in robotsparser with a RFC-conformant implementation that supports modern robots.txt constructs like Sitemaps, Allow, and Crawl-delay. Main features:

  • Memoization of fetched robots.txt
  • Expiration taken from the Expires header
  • Batch queries
  • Configurable user agent for fetching robots.txt
  • Automatic refetching based on expiration
 
File Type Py Version Uploaded on Size
reppy-0.4.7.tar.gz (md5) Source 2017-06-20 73KB