A python package for NLP explainability
Project description
A python package for benchmarking interpretability techniques.
Free software: MIT license
Documentation: https://ferret.readthedocs.io.
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from ferret import Benchmark
model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
bench = Benchmark(model, tokenizer)
explanations = bench.explain("You look stunning!")
evaluations = bench.evaluate_explanations(explanations)
print(evaluations)
Features
ferret builds on top of the transformers library. The library supports explanations using:
Gradients
Integrated Gradinets
Gradient x Input word embeddings
SHAP
LIME
and evaluate explanations via:
Faithfulness measures.
AOPC Comprehensiveness
AOPC Sufficiency
Kendall’s tau correlation with leave-one-feature out
Plausibility measures.
AUPRC soft score plausibility
Token f1 hard score plausibility
Token IOU hard score plausibility
TODOs
Possibility to run on select device (“cpu”, “cuda”)
Sample-And-Occlusion explanations
Discretized Integrated Gradients: https://arxiv.org/abs/2108.13654
Visualization
bench = Benchmark(...)
explanations = ...
bench.show_table(explanations)
evaluations = bench.evaluate_explanations(explanations)
bench.show_evaluation_table(evaluations)
Datasets evaluations
bench = Benchmark(...)
hatexdata = bench.load_dataset("hatexplain")
dataset_explanations = bench.generate_dataset_explanations(hatexdata)
dataset_evaluations = bench.evaluate_dataset_explanations(dataset_explanations)
bench.show_dataset_evaluation_table(dataset_evaluations)
Credits
This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.
Cookiecutter: https://github.com/audreyr/cookiecutter
audreyr/cookiecutter-pypackage: https://github.com/audreyr/cookiecutter-pypackage
Logo and graphical assets made by Luca Attanasio.
History
0.1.0 (2022-05-30)
First release on PyPI.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for ferret_xai-0.2.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9931aac3d59282857f4b0d5a4c2642f687eabd7173e65f48ae22ad24fae686a9 |
|
MD5 | cecb88664568f399f8b87f2dab49c852 |
|
BLAKE2b-256 | 1bfde343c32e919fb948445e06f48711b9cc0138b0cba76bd072f76d6ab5e3bc |