skip to navigation
skip to content

Not Logged In

robotexclusionrulesparser 1.6.1

An alternative to Python's robotparser module

Latest Version: 1.6.2

Robotexclusionrulesparser is an alternative to the Python standard library module robotparser. It fetches and parses robots.txt files and can answer questions as to whether or not a given user agent is permitted to visit a certain URL.

This module has some features that the standard library module robotparser does not, including the ability to decode non-ASCII robots.txt files, respect for Expires headers and understanding of Crawl-delay and Sitemap directives and wildcard syntax in path names.

Complete documentation (including a comparison with the standard library module robotparser) is available in ReadMe.html.

Robotexclusionrulesparser is released under a BSD license.

 
  • Downloads (All Versions):
  • 1 downloads in the last day
  • 24 downloads in the last week
  • 259 downloads in the last month