Seamlessly extract the creation or modification date of web pages by scraping the HTML code or performing content guesses.
Project description
Description
Seamless extraction of the creation or modification date of web pages. htmldate provides following ways to date documents, based on HTML parsing and scraping functions:
Starting from the header of the page, it uses common patterns to identify date fields.
If this is not successful, it scans the whole document looking for structural markers.
If no date cue could be found, it finally runs a series of heuristics on the content (text and markup).
Usage
The module takes the HTML document as input (string format) and returns a date if a valid cue could be found in the document. The output string defaults to ISO 8601 YMD format.
According to the tests it should be compatible with all common versions of Python (2 & 3).
Installation
Install from package repository: pip install htmldate
Direct installation of the latest version over pip is possible (see build status):
pip install git+https://github.com/adbar/htmldate.git
Command-line
A basic command-line interface is included:
$ wget -qO- "http://blog.python.org/2016/12/python-360-is-now-available.html" | htmldate
2016-12-23Usage:
$ htmldate --help
htmldate [-h] [-v] [-s]
optional arguments:
-h, --help show this help message and exit
-v, --verbose increase output verbosity
-s, --safe safe mode: markup search onlyWithin Python
All the functions of the module are currently bundled in htmldate, the examples below use the external module requests.
In case the web page features clear metadata in the header, the extraction is straightforward:
>>> import requests
>>> import htmldate
>>> r = requests.get('https://www.theguardian.com/politics/2016/feb/17/merkel-eu-uk-germany-national-interest-cameron-justified')
>>> htmldate.find_date(r.text)
'2016-02-17'A more advanced analysis of the document structure is sometimes needed:
>>> r = requests.get('http://blog.python.org/2016/12/python-360-is-now-available.html')
>>> htmldate.find_date(r.text)
'# DEBUG analyzing: <h2 class="date-header"><span>Friday, December 23, 2016</span></h2>'
'# DEBUG result: 2016-12-23'
'2016-12-23'In the worst case, the module resorts to a guess based on an extensive search, which can be deactivated:
>>> r = requests.get('https://creativecommons.org/about/')
>>> htmldate.find_date(r.text)
'2017-08-11' # has been updated since
>>> htmldate.find_date(r.text, extensive_search=False)
>>>It is also possible to use already parsed HTML (i.e. a LXML tree object):
>>> from lxml import html
>>> mytree = html.fromstring('<html><body><span class="entry-date">July 12th, 2016</span></body></html>')
>>> htmldate.find_date(mytree)
'2016-07-12'The output format of the dates found can be set in a format known to Python’s datetime module, the default being %Y-%m-%d:
>>> r = requests.get('https://www.gnu.org/licenses/gpl-3.0.en.html')
>>> htmldate.find_date(r.text)
'2016-11-18'
>>> htmldate.find_date(r.text, outputformat='%d %B %Y')
'18 November 2016'There are however pages for which no date can be found, ever:
>>> r = requests.get('https://example.com')
>>> htmldate.find_date(r.text)
>>>Tests
A series of webpages triggering different structural and content patterns is included for testing purposes:
$ python tests/unit_tests.pyAdditional information
Context
There are web pages for which neither the URL nor the server response provide a reliable way to date the document, i.e. find when it was first published and/or last modified.
This module is part of methods to derive metadata from web documents in order to build text corpora for (computational) linguistic analysis. For more information:
Barbaresi, Adrien. “Efficient construction of metadata-enhanced web corpora”, Proceedings of the 10th Web as Corpus Workshop (WAC-X), 2016.
Kudos to…
dateparser (although it’s is still a bit slow)
A few patterns are derived from python-goose, metascraper, newspaper and articleDateExtractor. This module extends their coverage and robustness significantly.
Further analyses
If the date is nowhere to be found, it might be worth considering carbon dating the web page, however this is computationally expensive.
Pull requests are welcome.
Contact
See my contact page for details.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for htmldate-0.2.2-py2.py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | a94b77b65b3f24034a860d3848cfb1b8368faff0db59fa3ebd226f74ce65d707 |
|
| MD5 | 8767768d95c71215370fd5569545a38a |
|
| BLAKE2b-256 | 4dfa17f023bee70e65f0f47224eef3682498f975f12f5545966cac0730c2e38a |
