skip to navigation
skip to content

reppy 0.3.0

Replacement robots.txt Parser

Latest Version: 0.3.1

Replaces the built-in robotsparser with a RFC-conformant implementation that supports modern robots.txt constructs like Sitemaps, Allow, and Crawl-delay. Main features:

  • Memoization of fetched robots.txt
  • Expiration taken from the Expires header
  • Batch queries
  • Configurable user agent for fetching robots.txt
  • Automatic refetching basing on expiration
File Type Py Version Uploaded on Size
reppy-0.3.0.tar.gz (md5) Source 2014-08-11 7KB
  • Downloads (All Versions):
  • 110 downloads in the last day
  • 396 downloads in the last week
  • 1897 downloads in the last month