skip to navigation
skip to content

spider.py 0.5

Multithreaded crawling, reporting, and mirroring for Web and FTP

This module provides multithreaded crawling, reporting, and mirroring for Web and FTP in one convenient library. Crawling depth, maximum number of URLs to crawl, and maximum number of threads are user-configurable. Reports can be generated on external URLS, internal redirects to outside URLs, unparsable HTML, non-HTTP/FTP URLs, and broken links.