TLDR: TuneQn is a new utility that helps optimize ONNX deep neural network models by selectively applying quantization, a process that reduces model size and computational demands. It intelligently identifies which parts of a model to quantize to minimize accuracy loss while achieving significant reductions in model size, making AI models more efficient for deployment on various hardware.
In the rapidly evolving world of Artificial Intelligence, Deep Neural Networks (DNNs) have become indispensable, powering everything from image recognition to content generation. However, these powerful models often come with a significant drawback: they are large and computationally intensive, making them challenging to deploy on devices with limited resources.
To tackle this, a technique called quantization has gained popularity. In simple terms, quantization reduces the precision of a model’s parameters, like weights and biases, from high-precision formats (e.g., 32-bit floating point) to lower-precision formats (e.g., 8-bit integers). This process leads to smaller model sizes, faster inference times, and lower power consumption. The catch? It often results in a decrease in prediction accuracy.
While some accuracy loss is usually acceptable, it must remain within tolerable limits for practical applications. When the accuracy degradation from full quantization becomes too high, developers often turn to selective quantization. This approach involves quantizing only a subset of layers or assigning different bit widths to various layers, helping to keep accuracy loss in check. However, determining which layers to quantize and to what extent has been a complex challenge, lacking a systematic approach to balance accuracy, model size, and performance.
Introducing TuneQn: A Smart Solution for Selective Quantization
Addressing these challenges, researchers Nikolaos Louloudakis and Ajitha Rajan from the University of Edinburgh have proposed TuneQn, a novel utility designed for selectively quantizing ONNX (Open Neural Network Exchange) models. TuneQn offers a comprehensive suite for selective quantization, deployment, and execution of ONNX models across various CPU and GPU devices. It integrates profiling and multi-objective optimization to identify the best model candidates.
TuneQn operates through an intuitive workflow. It starts by loading models and analyzing their layers to identify those most susceptible to errors when quantized. It then generates selectively quantized ONNX models by progressively exempting layers from quantization. For each configuration, TuneQn measures performance metrics such as accuracy and model size. Finally, it employs Pareto Front minimization, a multi-objective optimization technique, to pinpoint the optimal model candidates that strike the best balance between these objectives, and visualizes the results for users.
How TuneQn Works Under the Hood
TuneQn is built around five core modules:
- Model Orchestrator: Handles fetching, loading, and preparing models for quantization and execution, supporting models from local storage or online repositories like the ONNX Model Hub.
- Selective Quantization Module: Utilizes the ONNX Quantizer to create selectively quantized models. It supports both Static Quantization (which uses calibration data for better accuracy) and Dynamic Quantization (which quantizes activations at runtime, suitable for transformer models).
- Layer Activation Analysis Module: Identifies layers most affected by quantization by comparing activations from original and fully quantized models. It calculates ‘QDQ Error’ (simulating quantization effects) and ‘XModel Error’ (actual relative error introduced by each layer) to rank layers for selective quantization.
- Runner Module: Executes both original and quantized models on a given dataset. It supports deployment on CPUs using ONNX Runtime and on GPUs (including Android devices) using Apache TVM.
- Model Benchmarking Module: Collects and evaluates data after model execution, calculating accuracy differences and extracting model size information. It then applies Pareto Front analysis to select the top optimal solutions, saving results in a JSON report.
- Objectives Visualizer Module: Provides visual representations of activation errors across layers and plots of objectives, including the Pareto Fronts, to help users understand the trade-offs and identify the best model candidate.
Also Read:
- Improving LLM Compression: A New Method for Accurate Quantization
- OrQstrator: An AI-Driven Approach to Quantum Circuit Optimization for NISQ Devices
Impressive Results and Future Outlook
To demonstrate its effectiveness, TuneQn was evaluated on four well-known classification models: MobileNetV2, ShuffleNetV2, EfficientNet-Lite4, and ResNet50. These models were tested on both an Intel-i5 CPU and a Mali-G68 MC4 GPU, using both Static and Dynamic quantization settings.
The results were compelling. TuneQn successfully identified model candidates that achieved up to a 54.14% reduction in accuracy loss compared to fully quantized models. Furthermore, it selected models with significantly smaller sizes, demonstrating up to a 72.9% reduction in model size compared to the original, unquantized models. The researchers noted that the optimal Pareto Front results varied depending on the hardware device and the specific model, highlighting the importance of hardware-aware tuning.
The development team plans to expand TuneQn’s capabilities in future work, including support for more model architectures like object detectors (e.g., Tiny YOLOv3, SSD) and transformers (e.g., T5, GPT-2). This will involve integrating additional metrics like F1 score and BLEU score into the Pareto Front analysis. They also aim to refine execution time measurements and conduct experiments on GPU devices with limited support for specific quantized bit widths (e.g., INT8, INT4, mixed-precision operations).
TuneQn promises to be a valuable tool for machine learning developers and researchers, enabling them to systematically identify optimal ONNX model candidates that meet specific performance criteria by intelligently optimizing the selective quantization process. For more details, you can refer to the research paper.
