OpenVINO™ Integration with Optimum*

This module is an extension for Optimum* library which brings OpenVINO™ backend for Hugging Face Transformers* :hugs:.

This project provides multiple APIs to enable different tools:

• OpenVINO Runtime
• Neural Network Compression Framework (NNCF)

Install

Install only runtime:

pip install openvino-optimum

or with all dependencies (nncf and openvino-dev):

pip install openvino-optimum[all]

OpenVINO Runtime

This module provides an inference API for Hugging Face models. There are options to use models with PyTorch*, TensorFlow* pretrained weights or use native OpenVINO IR format (a pair of files ov_model.xml and ov_model.bin).

To use OpenVINO backend, import one of the AutoModel classes with OV prefix. Specify a model name or local path in from_pretrained method.

from optimum.intel.openvino import OVAutoModel

# PyTorch trained model with OpenVINO backend
model = OVAutoModel.from_pretrained(<name_or_path>, from_pt=True)

# TensorFlow trained model with OpenVINO backend
model = OVAutoModel.from_pretrained(<name_or_path>, from_tf=True)

# Initialize a model from OpenVINO IR
model = OVAutoModel.from_pretrained(<name_or_path>)

NNCF

NNCF is used for model training with applying such features like quantization, pruning. To enable NNCF in your training pipeline do the following steps:

1. Import NNCFAutoConfig:

from optimum.intel.nncf import NNCFAutoConfig

NOTE: NNCFAutoConfig must be imported before transformers to make magic work

2. Initialize a config from .json file:

nncf_config = NNCFAutoConfig.from_json(training_args.nncf_config)

3. Pass a config to Trainer object. In example,

model = AutoModelForQuestionAnswering.from_pretrained(<name_op_path>)

...

trainer = QuestionAnsweringTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset if training_args.do_train else None,
    eval_dataset=eval_dataset if training_args.do_eval else None,
    eval_examples=eval_examples if training_args.do_eval else None,
    tokenizer=tokenizer,
    data_collator=data_collator,
    post_process_function=post_processing_function,
    compute_metrics=compute_metrics,
    nncf_config=nncf_config,
)

NNCF module is independent from the Runtime module so model class should not be wrapped to one of OVAutoModel classes.

Training examples can be found in Transformers library. NNCF configs are published in config folder. Add --nncf_config with a path to corresponding config when training your model. More command line examples here.

python examples/pytorch/token-classification/run_ner.py --model_name_or_path bert-base-cased --dataset_name conll2003 --output_dir bert_base_cased_conll_int8 --do_train --do_eval --save_strategy epoch --evaluation_strategy epoch --nncf_config nncf_bert_config_conll.json

To use the NNCF component, install the package with [nncf] or [all] extras:

pip install openvino-optimum[nncf]

See the Changelog page for details about module development.

*Other names and brands may be claimed as the property of others.