Fast and robust extraction of original and updated publication dates from URLs and web pages.
Project description
Find original and updated publication dates of any web page. On the command-line or with Python, all the steps needed from web page download to HTML parsing, scraping, and text analysis are included. The package is used in production on millions of documents and integrated by multiple libraries.
In a nutshell
With Python
>>> from htmldate import find_date
>>> find_date('http://blog.python.org/2016/12/python-360-is-now-available.html')
'2016-12-23'On the command-line
$ htmldate -u http://blog.python.org/2016/12/python-360-is-now-available.html
'2016-12-23'Features
Flexible input: URLs, HTML files, or HTML trees can be used as input (including batch processing).
Customizable output: Any date format (defaults to ISO 8601 YMD).
Detection of both original and updated dates.
Multilingual.
Compatible with all recent versions of Python.
How it works
Htmldate operates by sifting through HTML markup and if necessary text elements. It features the following heuristics:
Markup in header: Common patterns are used to identify relevant elements (e.g. link and meta elements) including Open Graph protocol attributes.
HTML code: The whole document is searched for structural markers like abbr or time elements and a series of attributes (e.g. postmetadata).
Bare HTML content: Heuristics are run on text and markup: - In fast mode the HTML page is cleaned and precise patterns are targeted. - In extensive mode all potential dates are collected and a disambiguation algorithm determines the best one.
Finally, the output is validated and converted to the chosen format.
Performance
1000 web pages containing identifiable dates (as of 2023-11-13 on Python 3.10) |
|||||
|---|---|---|---|---|---|
Python Package |
Precision |
Recall |
Accuracy |
F-Score |
Time |
articleDateExtractor 0.20 |
0.803 |
0.734 |
0.622 |
0.767 |
5x |
date_guesser 2.1.4 |
0.781 |
0.600 |
0.514 |
0.679 |
18x |
goose3 3.1.17 |
0.869 |
0.532 |
0.493 |
0.660 |
15x |
htmldate[all] 1.6.0 (fast) |
0.883 |
0.924 |
0.823 |
0.903 |
1x |
htmldate[all] 1.6.0 (extensive) |
0.870 |
0.993 |
0.865 |
0.928 |
1.7x |
newspaper3k 0.2.8 |
0.769 |
0.667 |
0.556 |
0.715 |
15x |
news-please 1.5.35 |
0.801 |
0.768 |
0.645 |
0.784 |
34x |
For the complete results and explanations see evaluation page.
Installation
Htmldate is tested on Linux, macOS and Windows systems, it is compatible with Python 3.6 upwards. It can notably be installed with pip (pip3 where applicable) from the PyPI package repository:
pip install htmldate
(optionally) pip install htmldate[speed]
Documentation
For more details on installation, Python & CLI usage, please refer to the documentation: htmldate.readthedocs.io
License
This package is distributed under the Apache 2.0 license.
Versions prior to v1.8.0 are under GPLv3+ license.
Author
This project is part of methods to derive information from web documents in order to build text databases for research (chiefly linguistic analysis and natural language processing).
Extracting and pre-processing web texts to meet the exacting standards is a significant challenge. It is often not possible to reliably determine the date of publication or modification using either the URL or the server response. For more information:
@article{barbaresi-2020-htmldate,
title = {{htmldate: A Python package to extract publication dates from web pages}},
author = "Barbaresi, Adrien",
journal = "Journal of Open Source Software",
volume = 5,
number = 51,
pages = 2439,
url = {https://doi.org/10.21105/joss.02439},
publisher = {The Open Journal},
year = 2020,
}Barbaresi, A. “htmldate: A Python package to extract publication dates from web pages”, Journal of Open Source Software, 5(51), 2439, 2020. DOI: 10.21105/joss.02439
Barbaresi, A. “Generic Web Content Extraction with Open-Source Software”, Proceedings of KONVENS 2019, Kaleidoscope Abstracts, 2019.
Barbaresi, A. “Efficient construction of metadata-enhanced web corpora”, Proceedings of the 10th Web as Corpus Workshop (WAC-X), 2016.
You can contact me via my contact page or GitHub.
Contributing
Contributions are welcome as well as issues filed on the dedicated page.
Special thanks to the contributors who have submitted features and bugfixes!
Acknowledgements
Kudos to the following software libraries:
A few patterns are derived from the python-goose, metascraper, newspaper and articleDateExtractor libraries. This module extends their coverage and robustness significantly.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
