skip to navigation
skip to content

robotexclusionrulesparser 1.6.2

A robots.txt parser alternative to Python's robotparser module

Latest Version: 1.7.1

Robotexclusionrulesparser is an alternative to the Python standard library module robotparser. It fetches and parses robots.txt files and can answer questions as to whether or not a given user agent is permitted to visit a certain URL.

This module has some features that the standard library module robotparser does not, including the ability to decode non-ASCII robots.txt files, respect for Expires headers and understanding of Crawl-delay and Sitemap directives and wildcard syntax in path names.

Complete documentation (including a comparison with the standard library module robotparser) is available in ReadMe.html.

Robotexclusionrulesparser is released under a BSD license.

File Type Py Version Uploaded on Size
robotexclusionrulesparser-1.6.2.tar.gz (md5) Source 2014-03-25 26KB