pyrobotstxt: Python Package for robots.txt Files

pyrobotstxt package can be used to (systematically) generate robots.txt files. Moreover, this package comes in handy for creating and including ASCII images in robots.txt files.

In future releases, it would be possible to parse and analyze robots.txt file generated using any software (not limited to pyrobotstxt)

Whats in pyrobotstxt?

We believe in monolithic software development and created this tiny package that does its job without any bloat. It is useful for

• Createing robots.txt File
• Parsing a robots.txt File [in progress]
• Analyzing robots.txt File [in progress]

How to Use pyrobotstxt?

You can find detailed tutorial on pyrobotstxt tutorial website. You can consult pyrobotstxt documentation website for API reference.

Contribute

Pull Requests, Feature Suggestions, and collaborations are welcome.

About Us

This work is a collaborative effort of seowings, and serpwings.