A python package for NLP explainability
Project description
A python package for benchmarking interpretability techniques.
Free software: MIT license
Documentation: https://ferret.readthedocs.io.
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from ferret import Benchmark
model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
bench = Benchmark(model, tokenizer)
explanations = bench.explain("You look stunning!")
evaluations = bench.evaluate_explanations(explanations)
print(evaluations)
Features
ferret builds on top of the transformers library. The library supports explanations using:
Gradients
Integrated Gradinets
Gradient x Input word embeddings
SHAP
LIME
and evaluate explanations via:
Faithfulness measures.
AOPC Comprehensiveness
AOPC Sufficiency
Kendall’s tau correlation with leave-one-feature out
Plausibility measures.
AUPRC soft score plausibility
Token f1 hard score plausibility
Token IOU hard score plausibility
TODOs
Possibility to run on select device (“cpu”, “cuda”)
Sample-And-Occlusion explanations
Discretized Integrated Gradients: https://arxiv.org/abs/2108.13654
Visualization
bench = Benchmark(...)
explanations = ...
bench.show_table(explanations)
evaluations = bench.evaluate_explanations(explanations)
bench.show_evaluation_table(evaluations)
Datasets evaluations
bench = Benchmark(...)
hatexdata = bench.load_dataset("hatexplain")
sample_evaluations = bench.evaluate_samples(hatexdata, [1,2])
bench.show_samples_evaluation_table(sample_evaluations)
Credits
This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.
Cookiecutter: https://github.com/audreyr/cookiecutter
audreyr/cookiecutter-pypackage: https://github.com/audreyr/cookiecutter-pypackage
Logo and graphical assets made by Luca Attanasio.
History
0.1.0 (2022-05-30)
First release on PyPI.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for ferret_xai-0.3.2-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 274b85c9385e9c68b0f4e7680af911be0ccb05639ae582ca471f1ba2d1ef235f |
|
MD5 | 863eada37428f7ae30d52619994144ea |
|
BLAKE2b-256 | e37982a927d375829b5c06cdd7ad230bfd033dd772968c2234e643c330a82ca0 |