courlan 1.0.0

check_url()

All useful operations chained in check_url(url):

>>> from courlan import check_url

# return url and domain name
>>> check_url('https://github.com/adbar/courlan')
('https://github.com/adbar/courlan', 'github.com')

# filter out bogus domains
>>> check_url('http://666.0.0.1/')
>>>

# tracker removal
>>> check_url('http://test.net/foo.html?utm_source=twitter#gclid=123')
('http://test.net/foo.html', 'test.net')

# use strict for further trimming
>>> my_url = 'https://httpbin.org/redirect-to?url=http%3A%2F%2Fexample.org'
>>> check_url(my_url, strict=True)
('https://httpbin.org/redirect-to', 'httpbin.org')

# check for redirects (HEAD request)
>>> url, domain_name = check_url(my_url, with_redirects=True)

# include navigation pages instead of discarding them
>>> check_url('http://www.example.org/page/10/', with_nav=True)

# remove trailing slash
>>> check_url('https://github.com/adbar/courlan/', trailing_slash=False)

Language-aware heuristics, notably internationalization in URLs, are available in lang_filter(url, language):

# optional language argument
>>> url = 'https://www.un.org/en/about-us'

# success: returns clean URL and domain name
>>> check_url(url, language='en')
('https://www.un.org/en/about-us', 'un.org')

# failure: doesn't return anything
>>> check_url(url, language='de')
>>>

# optional argument: strict
>>> url = 'https://en.wikipedia.org/'
>>> check_url(url, language='de', strict=False)
('https://en.wikipedia.org', 'wikipedia.org')
>>> check_url(url, language='de', strict=True)
>>>

Define stricter restrictions on the expected content type with strict=True. Also blocks certain platforms and pages types crawlers should stay away from if they don’t target them explicitly and other black holes where machines get lost.

# strict filtering: blocked as it is a major platform
>>> check_url('https://www.twitch.com/', strict=True)
>>>

Sampling by domain name

>>> from courlan import sample_urls
>>> my_urls = ['https://example.org/' + str(x) for x in range(100)]
>>> my_sample = sample_urls(my_urls, 10)
# optional: exclude_min=None, exclude_max=None, strict=False, verbose=False

Web crawling and URL handling

Determine if a link leads to another host:

>>> from courlan import is_external
>>> is_external('https://github.com/', 'https://www.microsoft.com/')
True
# default
>>> is_external('https://google.com/', 'https://www.google.co.uk/', ignore_suffix=True)
False
# taking suffixes into account
>>> is_external('https://google.com/', 'https://www.google.co.uk/', ignore_suffix=False)
True

Other useful functions dedicated to URL handling:

• extract_domain(url, fast=True): find domain and subdomain or just domain with fast=False

• get_base_url(url): strip the URL of some of its parts

• get_host_and_path(url): decompose URLs in two parts: protocol + host/domain and path

• get_hostinfo(url): extract domain and host info (protocol + host/domain)

• fix_relative_urls(baseurl, url): prepend necessary information to relative links

>>> from courlan import *
>>> url = 'https://www.un.org/en/about-us'

>>> get_base_url(url)
'https://www.un.org'

>>> get_host_and_path(url)
('https://www.un.org', '/en/about-us')

>>> get_hostinfo(url)
('un.org', 'https://www.un.org')

>>> fix_relative_urls('https://www.un.org', 'en/about-us')
'https://www.un.org/en/about-us'

Other filters dedicated to crawl frontier management:

• is_not_crawlable(url): check for deep web or pages generally not usable in a crawling context

• is_navigation_page(url): check for navigation and overview pages

>>> from courlan import is_navigation_page, is_not_crawlable
>>> is_navigation_page('https://www.randomblog.net/category/myposts')
True
>>> is_not_crawlable('https://www.randomblog.net/login')
True

Python helpers

Helper function, scrub and normalize:

>>> from courlan import clean_url
>>> clean_url('HTTPS://WWW.DWDS.DE:80/')
'https://www.dwds.de'

Basic scrubbing only:

>>> from courlan import scrub_url

Basic canonicalization/normalization only, i.e. modifying and standardizing URLs in a consistent manner:

>>> from urllib.parse import urlparse
>>> from courlan import normalize_url
>>> my_url = normalize_url(urlparse(my_url))
# passing URL strings directly also works
>>> my_url = normalize_url(my_url)
# remove unnecessary components and re-order query elements
>>> normalize_url('http://test.net/foo.html?utm_source=twitter&post=abc&page=2#fragment', strict=True)
'http://test.net/foo.html?page=2&post=abc'

Basic URL validation only:

>>> from courlan import validate_url
>>> validate_url('http://1234')
(False, None)
>>> validate_url('http://www.example.org/')
(True, ParseResult(scheme='http', netloc='www.example.org', path='/', params='', query='', fragment=''))

Troubleshooting

Courlan uses an internal cache to speed up URL parsing. It can be reset as follows:

>>> from courlan.meta import clear_caches
>>> clear_caches()

UrlStore class

The UrlStore class allow for storing and retrieving domain-classified URLs, where a URL like https://example.org/path/testpage is stored as the path /path/testpage within the domain https://example.org. It features the following methods:

• URL management

  • add_urls(urls=[], appendleft=None, visited=False): Add a list of URLs to the (possibly) existing one. Optional: append certain URLs to the left, specify if the URLs have already been visited.

  • add_from_html(htmlstring, url, external=False, lang=None, with_nav=True): Extract and filter links in a HTML string.

  • discard(domains): Declare domains void and prune the store.

  • dump_urls(): Return a list of all known URLs.

  • print_urls(): Print all URLs in store (URL + TAB + visited or not).

  • print_unvisited_urls(): Print all unvisited URLs in store.

  • get_all_counts(): Return all download counts for the hosts in store.

  • get_known_domains(): Return all known domains as a list.

  • get_unvisited_domains(): Find all domains for which there are unvisited URLs.

  • total_url_number(): Find number of all URLs in store.

  • is_known(url): Check if the given URL has already been stored.

  • has_been_visited(url): Check if the given URL has already been visited.

  • filter_unknown_urls(urls): Take a list of URLs and return the currently unknown ones.

  • filter_unvisited_urls(urls): Take a list of URLs and return the currently unvisited ones.

  • find_known_urls(domain): Get all already known URLs for the given domain (ex. “https://example.org”).

  • find_unvisited_urls(domain): Get all unvisited URLs for the given domain.

  • get_unvisited_domains(): Return all domains which have not been all visited.

  • reset(): Re-initialize the URL store.

• Crawling and downloads

  • get_url(domain): Retrieve a single URL and consider it to be visited (with corresponding timestamp).

  • get_rules(domain): Return the stored crawling rules for the given website.

  • store_rules(website, rules=None): Store crawling rules for a given website.

  • get_crawl_delay(): Return the delay as extracted from robots.txt, or a given default.

  • get_download_urls(timelimit=10): Get a list of immediately downloadable URLs according to the given time limit per domain.

  • establish_download_schedule(max_urls=100, time_limit=10): Get up to the specified number of URLs along with a suitable backoff schedule (in seconds).

  • download_threshold_reached(threshold): Find out if the download limit (in seconds) has been reached for one of the websites in store.

  • unvisited_websites_number(): Return the number of websites for which there are still URLs to visit.

  • is_exhausted_domain(domain): Tell if all known URLs for the website have been visited.

• Persistance

  • write(filename): Save the store to disk.

  • load_store(filename): Read a UrlStore from disk (separate function, not class method).

Optional settings: - compressed=True: activate compression of URLs and rules - language=XX: focus on a particular target language (two-letter code) - strict=True: stricter URL filtering - verbose=True: dump URLs if interrupted (requires use of signal)