Repository of Intel® Neural Compressor
Project description
Intel® Neural Compressor
An open-source Python library supporting popular model compression techniques on all mainstream deep learning frameworks (TensorFlow, PyTorch, ONNX Runtime, and MXNet)
Architecture | Workflow | Results | Examples | Documentations
Intel® Neural Compressor aims to provide popular model compression techniques such as quantization, pruning (sparsity), distillation, and neural architecture search on mainstream frameworks such as TensorFlow, PyTorch, ONNX Runtime, and MXNet, as well as Intel extensions such as Intel Extension for TensorFlow and Intel Extension for PyTorch. In particular, the tool provides the key features, typical examples, and open collaborations as below:
-
Support a wide range of Intel hardware such as Intel Xeon Scalable Processors, Intel Xeon CPU Max Series, Intel Data Center GPU Flex Series, and Intel Data Center GPU Max Series with extensive testing; support AMD CPU, ARM CPU, and NVidia GPU through ONNX Runtime with limited testing
-
Validate popular LLMs such as LLama2, LLama, MPT, Falcon, GPT-J, Bloom, OPT, and more than 10,000 broad models such as Stable Diffusion, BERT-Large, and ResNet50 from popular model hubs such as Hugging Face, Torch Vision, and ONNX Model Zoo, by leveraging zero-code optimization solution Neural Coder and automatic accuracy-driven quantization strategies
-
Collaborate with cloud marketplace such as Google Cloud Platform, Amazon Web Services, and Azure, software platforms such as Alibaba Cloud, Tencent TACO and Microsoft Olive, and open AI ecosystem such as Hugging Face, PyTorch, ONNX, ONNX Runtime, and Lightning AI
Installation
Install from pypi
pip install neural-compressor
More installation methods can be found at Installation Guide. Please check out our FAQ for more details.
Getting Started
Quantization with Python API
# Install Intel Neural Compressor and TensorFlow
pip install neural-compressor
pip install tensorflow
# Prepare fp32 model
wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_6/mobilenet_v1_1.0_224_frozen.pb
from neural_compressor.data import DataLoader, Datasets
from neural_compressor.config import PostTrainingQuantConfig
dataset = Datasets("tensorflow")["dummy"](shape=(1, 224, 224, 3))
dataloader = DataLoader(framework="tensorflow", dataset=dataset)
from neural_compressor.quantization import fit
q_model = fit(
model="./mobilenet_v1_1.0_224_frozen.pb",
conf=PostTrainingQuantConfig(),
calib_dataloader=dataloader,
)
Documentation
| Overview | |||||||
|---|---|---|---|---|---|---|---|
| Architecture | Workflow | Examples | APIs | ||||
| Python-based APIs | |||||||
| Quantization | Advanced Mixed Precision | Pruning (Sparsity) | Distillation | ||||
| Orchestration | Benchmarking | Distributed Compression | Model Export | ||||
| Neural Coder (Zero-code Optimization) | |||||||
| Launcher | JupyterLab Extension | Visual Studio Code Extension | Supported Matrix | ||||
| Advanced Topics | |||||||
| Adaptor | Strategy | Distillation for Quantization | SmoothQuant | ||||
| Weight-Only Quantization (INT8/INT4/FP4/NF4) | FP8 Quantization | ||||||
| Innovations for Productivity | |||||||
| Neural Insights | Neural Solution | ||||||
More documentations can be found at User Guide.
Selected Publications/Events
- EMNLP'2023 (Under Review): TEQ: Trainable Equivalent Transformation for Quantization of LLMs (Sep 2023)
- arXiv: Efficient Post-training Quantization with FP8 Formats (Sep 2023)
- arXiv: Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs (Sep 2023)
- Post on Social Media: ONNXCommunityMeetup2023: INT8 Quantization for Large Language Models with Intel Neural Compressor (July 2023)
- Blog by Intel: Accelerate Llama 2 with Intel AI Hardware and Software Optimizations (July 2023)
- NeurIPS'2022: Fast Distilbert on CPUs (Oct 2022)
- NeurIPS'2022: QuaLA-MiniLM: a Quantized Length Adaptive MiniLM (Oct 2022)
View Full Publication List.
Additional Content
Communication
- GitHub Issues: mainly for bugs report, new feature request, question asking, etc.
- Email: welcome to raise any interesting research ideas on model compression techniques by email for collaborations.
- WeChat group: scan the QA code to join the technical discussion.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for neural_compressor-2.3.1-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | d94ba25aad77289c24d475a1967889832a9712ba685afa03093a06d93b84d1d0 |
|
| MD5 | 32a133687c8cda7ce64598061114b83c |
|
| BLAKE2b-256 | 971cc6b3917ae047402f4749a5e2ec8c398687120fb7100def968f403f74c873 |
