A python package for NLP explainability
Project description
A python package for benchmarking interpretability techniques.
Free software: MIT license
Documentation: https://ferret.readthedocs.io.
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from ferret import Benchmark
model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
bench = Benchmark(model, tokenizer)
explanations = bench.explain("You look stunning!")
evaluations = bench.evaluate_explanations(explanations)
print(evaluations)
Features
ferret builds on top of the transformers library. The library supports explanations using:
Gradients
Integrated Gradinets
Gradient x Input word embeddings
SHAP
LIME
and evaluate explanations via:
Faithfulness measures.
AOPC Comprehensiveness
AOPC Sufficiency
Kendall’s tau correlation with leave-one-feature out
Plausibility measures.
AUPRC soft score plausibility
Token f1 hard score plausibility
Token IOU hard score plausibility
TODOs
Possibility to run on select device (“cpu”, “cuda”)
Sample-And-Occlusion explanations
Discretized Integrated Gradients: https://arxiv.org/abs/2108.13654
Visualization
bench = Benchmark(...)
explanations = ...
bench.show_table(explanations)
evaluations = bench.evaluate_explanations(explanations)
bench.show_evaluation_table(evaluations)
Datasets evaluations
bench = Benchmark(...)
hatexdata = bench.load_dataset("hatexplain")
sample_evaluations = bench.evaluate_samples(hatexdata, [1,2])
bench.show_samples_evaluation_table(sample_evaluations)
Credits
This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.
Cookiecutter: https://github.com/audreyr/cookiecutter
audreyr/cookiecutter-pypackage: https://github.com/audreyr/cookiecutter-pypackage
Logo and graphical assets made by Luca Attanasio.
History
0.1.0 (2022-05-30)
First release on PyPI.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for ferret_xai-0.3.3-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fb00edf4a63d34a9d1236716abb8a2dce0515b54ce27f0e6ddd3916652abbd6a |
|
MD5 | 5b382e78f71039ead5aa23f84a8dc248 |
|
BLAKE2b-256 | 83cc55e29453da37f258e965d0dd27a2285c5c1c604ebf12e5ac01f89e6cede4 |