TextDescriptives

A Python package for calculating a large variety of statistics from text(s).

Installation

Clone the Github directory, navigate to it in a terminal, and call pip install .

Usage

To calculate all possible metrics:

import textdescriptives

# Input can be either a string, list of strings, or pandas Series 
en_test = ['The world is changed. I feel it in the water. I feel it in the earth. I smell it in the air. Much that once was is lost, for none now live who remember it.',
            'He felt that his whole life was some kind of dream and he sometimes wondered whose it was and whether they were enjoying it.']

textdescriptives.all_metrics(en_test, lang = 'en', snlp_path = snlp_path)

To calculate one category at a time:

textdescriptives.basic_stats(texts, lang = 'en', metrics = 'all')
textdescriptives.readability(texts, lang = 'en')
textdescriptives.etymology(texts, lang = 'en')
textdescriptives.dependency_distance(texsts, lang = 'en', snlp_path = None)

Textdescriptives works for most languages, simply change the country code:

da_test = pd.Series(['Da jeg var atten, tog jeg patent på ild. Det skulle senere vise sig at blive en meget indbringende forretning',
            "Spis skovsneglen, Mulle. Du vil jo gerne være med i hulen, ikk'?"])

textdescriptives.all_metrics(da_test, lang = 'da', snlp_path=snlp_path)

If you only want a subset of the basic statistics

textdescriptives.basic_stats(en_test, lang = 'en', metrics=['avg_word_length', 'n_chars'])

Readability

The readability measures are largely derived from the textstat library and are thoroughly defined there.

Etymology

The etymology measures are calculated using macroetym only slightly rewritten to be called from a script. They are calculated since in English, a greater frequency of words with a Latinate origin tends to indicate a more formal language register.

Dependency Distance

Mean dependency distance can be used as a way of measuring the average syntactic complexity of a text. Requres the snlp library. The dependency distance function requires stanfordnlp, and their language models. If you have already downloaded these models, the path to the folder can be specified in the snlp_path paramter. Otherwise, the models will be downloaded to your working directory + /snlp_resources.

Dependencies

Depending on which measures you want to calculate, the dependencies differ.

• Basic and readability: numpy, pandas, pyphen, pycountry
• Etymology: nltk and the following models python3 -c "import nltk; nltk.download('punkt'); nltk.download('stopwords'); nltk.download('averaged_perceptron_tagger'); nltk.download('wordnet')"
• Depedency distance: snlp

Metrics

Metrics currently implemented:

1. Basic descriptive statistics - mean, median, standard deviation of the following:

• Word length
• Sentence length, words
• Sentence length, characters (TODO)
• Syllables per word
• Number of characters
• Number of sentences
• Number of types (unique words)
• Number of tokens (total words)
• Type/toḱen ratio
• Lix
• Rix

2. Readability metrics:

• Gunning-Fog
• SMOG
• Flesch reading ease
• Flesch-Kincaid grade
• Automated readability index
• Coleman-Liau index

3. Etymology-related metrics:

• Percentage words with Germanic origin
• Percentage words with Latinate origin
• Latinate/Germanic origin ratio

4. Dependency distance metrics:

• Mean dependency distance, sentence level (mean, standard deviation)
• Mean proportion adjacent dependency relations, sentence level (mean, standard devaiation)

Developed by Lasse Hansen at the Center for Humanities Computing Aarhus