A python package for NLP explainability
Project description
A python package for benchmarking interpretability techniques.
Free software: MIT license
Documentation: https://ferret.readthedocs.io.
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from ferret import Benchmark
model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
bench = Benchmark(model, tokenizer)
explanations = bench.explain("You look stunning!")
evaluations = bench.evaluate_explanations(explanations)
print(evaluations)
Features
ferret builds on top of the transformers library. The library supports explanations using:
Gradients
Integrated Gradinets
Gradient x Input word embeddings
SHAP
LIME
and evaluate explanations via:
Faithfulness measures.
AOPC Comprehensiveness
AOPC Sufficiency
Kendall’s tau correlation with leave-one-feature out
Plausibility measures.
AUPRC soft score plausibility
Token f1 hard score plausibility
Token IOU hard score plausibility
TODOs
Possibility to run on select device (“cpu”, “cuda”)
Sample-And-Occlusion explanations
Discretized Integrated Gradients: https://arxiv.org/abs/2108.13654
Visualization
bench = Benchmark(...)
explanations = ...
bench.show_table(explanations)
evaluations = bench.evaluate_explanations(explanations)
bench.show_evaluation_table(evaluations)
Datasets evaluations
bench = Benchmark(...)
hatexdata = bench.load_dataset("hatexplain")
sample_evaluations = bench.evaluate_samples(hatexdata, [1,2])
bench.show_samples_evaluation_table(sample_evaluations)
Credits
This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.
Cookiecutter: https://github.com/audreyr/cookiecutter
audreyr/cookiecutter-pypackage: https://github.com/audreyr/cookiecutter-pypackage
Logo and graphical assets made by Luca Attanasio.
History
0.1.0 (2022-05-30)
First release on PyPI.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for ferret_xai-0.3.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 17fae4109ab3530b24b3aed20f5895512094b3f5091863d14b1ee2bbe0f2c836 |
|
MD5 | 748e17c76eb0db736921c4694136a6f5 |
|
BLAKE2b-256 | 24c1fc06e27bc4dbeb48fd0f1ffa94a977890e05f0039640a9ab0b142f5ba19d |