skip to navigation
skip to content

reppy 0.3.0

Replacement robots.txt Parser

Latest Version: 0.4.9

Replaces the built-in robotsparser with a RFC-conformant implementation that supports modern robots.txt constructs like Sitemaps, Allow, and Crawl-delay. Main features:

  • Memoization of fetched robots.txt
  • Expiration taken from the Expires header
  • Batch queries
  • Configurable user agent for fetching robots.txt
  • Automatic refetching basing on expiration
File Type Py Version Uploaded on Size
reppy-0.3.0.tar.gz (md5) Source 2014-08-11 7KB