A python package for NLP explainability
Project description
A python package for benchmarking interpretability techniques.
Free software: MIT license
Documentation: https://ferret.readthedocs.io.
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from ferret import Benchmark
model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
bench = Benchmark(model, tokenizer)
explanations = bench.explain("You look stunning!")
evaluations = bench.evaluate_explanations(explanations)
print(evaluations)
Features
ferret builds on top of the transformers library. The library supports explanations using:
Gradients
Integrated Gradinets
Gradient x Input word embeddings
SHAP
LIME
and evaluate explanations via:
Faithfulness measures.
AOPC Comprehensiveness
AOPC Sufficiency
Kendall’s tau correlation with leave-one-feature out
Plausibility measures.
AUPRC soft score plausibility
Token f1 hard score plausibility
Token IOU hard score plausibility
TODOs
Possibility to run on select device (“cpu”, “cuda”)
Sample-And-Occlusion explanations
Discretized Integrated Gradients: https://arxiv.org/abs/2108.13654
Visualization
bench = Benchmark(...)
explanations = ...
bench.show_table(explanations)
evaluations = bench.evaluate_explanations(explanations)
bench.show_evaluation_table(evaluations)
Datasets evaluations
bench = Benchmark(...)
hatexdata = bench.load_dataset("hatexplain")
sample_evaluations = bench.evaluate_samples(hatexdata, [1,2])
bench.show_samples_evaluation_table(sample_evaluations)
Credits
This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.
Cookiecutter: https://github.com/audreyr/cookiecutter
audreyr/cookiecutter-pypackage: https://github.com/audreyr/cookiecutter-pypackage
Logo and graphical assets made by Luca Attanasio.
History
0.1.0 (2022-05-30)
First release on PyPI.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for ferret_xai-0.3.1-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d8db663832b2d6084b86ca5f6d692cf60bef544c7209ee03f00f84a922caec60 |
|
MD5 | 00c31e549a742dae1cabfdc498288014 |
|
BLAKE2b-256 | 00adf5fc6a26fcc11f87b890e30cb70edbf46250c104a07e9c433d6d53a29bd4 |