skip to navigation
skip to content

jusText 2.1.0

Heuristic based boilerplate removal tool

Latest Version: 2.2.0


Program jusText is a tool for removing boilerplate content, such as navigation links, headers, and footers from HTML pages. It is designed to preserve mainly text containing full sentences and it is therefore well suited for creating linguistic resources such as Web corpora. You can try it online.

This is a fork of original (currently unmaintained) code of jusText hosted on Google Code. Below are some alternatives that I found: Používají hustotu odkazů a délku odstavce - tak by mohli ještě přidat zanoření blokových prvků - a měli by náš prototyp z jyxa


Make sure you have Python 2.6+/3.2+ and pip (Windows, Linux) installed. Run simply (preferred way):

$ [sudo] pip install justext

Or for the fresh version:

$ [sudo] pip install git+git://

Or if you have to:

$ wget # download the sources
$ unzip # extract the downloaded file
$ jusText-master/
$ [sudo] python install # install the package




$ python -m justext -s Czech -o text.txt
$ python -m justext -s English -o plain_text.txt english_page.html
$ python -m justext --help # for more info

Python API

import requests
import justext

response = requests.get("")
paragraphs = justext.justext(response.content, justext.get_stoplist("English"))
for paragraph in paragraphs:
  if not paragraph.is_boilerplate:
    print paragraph.text


Run tests via

$ nosetests-2.6 && nosetests-3.2 && nosetests-2.7 && nosetests-3.3


This software has been developed at the Natural Language Processing Centre of Masaryk University in Brno with a financial support from PRESEMT and Lexical Computing Ltd. It also relates to PhD research of Jan Pomikálek.

Changelog for jusText

2.1.0 (2014-01-25)

  • FEATURE: Added XPath selector to the paragrahs. XPath selector is also available in detailed output as xpath attribute of <p> tag #5.

2.0.0 (2013-08-26)

  • FEATURE: Added pluggable DOM preprocessor.
  • FEATURE: Added support for Python 3.2+.
  • INCOMPATIBLE CHANGE: Paragraphs are instances of justext.paragraph.Paragraph.
  • INCOMPATIBLE CHANGE: Script ‘justext’ removed in favour of command python -m justext.
  • FEATURE: It’s possible to enter an URI as input document in CLI.
  • FEATURE: It is possible to pass unicode string directly.

1.2.0 (2011-08-08)

  • FEATURE: Character counts used instead of word counts where possible in order to make the algorithm work well in the language independent mode (without a stoplist) for languages where counting words is not easy (Japanese, Chinese, Thai, etc).
  • BUG FIX: More robust parsing of meta tags containing the information about used charset.
  • BUG FIX: Corrected decoding of HTML entities &#128; to &#159;

1.1.0 (2011-03-09)

  • First public release.
File Type Py Version Uploaded on Size
jusText-2.1.0-py2.py3-none-any.whl (md5) Python Wheel 2.7 2014-01-25 842KB
jusText-2.1.0.win32.exe (md5) MS Windows installer any 2014-01-26 1MB (md5) Source 2014-01-25 845KB