TLDR: Researchers have developed SQS, a new method that combines neural network pruning and quantization into a single process. This approach uses Bayesian learning with a “spike-and-slab” prior for sparsity and Gaussian Mixture Models for low-bit precision, allowing for much higher compression rates of large models like ResNet, BERT, and Llama, while maintaining strong performance.
Deploying powerful deep neural networks (DNNs) on devices with limited resources, such as smartphones or embedded systems, has always been a significant challenge. These networks, while achieving impressive results across various tasks, demand substantial computational power and memory. Traditionally, engineers have tackled this problem using two main strategies: weight pruning, which removes less important connections, and low-bit quantization, which reduces the precision of numerical representations. However, applying these methods separately often leads to compromises, where achieving high compression means sacrificing too much performance.
A new research paper introduces a groundbreaking solution called Sparse Quantized Sub-distributions (SQS), a unified framework that simultaneously prunes and quantizes neural networks. Developed by Ziyi Wang, Nan Jiang, Guang Lin, and Qifan Song, SQS leverages Bayesian variational learning to achieve superior compression rates while maintaining comparable performance to uncompressed models. This integrated approach is a significant step forward, as it avoids the suboptimal results often seen when pruning and quantization are applied in isolation.
The core innovation behind SQS lies in its unique approach to modeling network weights. It employs a “spike-and-slab” prior distribution. The “spike” component effectively encourages sparsity by pushing many weights to zero, thereby performing pruning. The “slab” component, on the other hand, models the remaining non-zero weights using Gaussian Mixture Models (GMMs). This GMM approach allows for low-bit precision, meaning weights can be represented using fewer bits, drastically reducing memory footprint and computational overhead.
By unifying these two compression techniques within a single variational learning process, SQS explores a broader and more effective solution space. Instead of making independent decisions about which weights to prune and how to quantize them, SQS learns a sparse, quantized sub-distribution over the network weights. This joint optimization helps mitigate the performance degradation typically associated with aggressive compression.
The researchers conducted extensive experiments to validate SQS across a range of widely used neural networks, including ResNet, BERT-base, Llama3, and Qwen2.5 models. The results were compelling: SQS consistently achieved higher compression rates than existing methods, requiring fewer parameters, while incurring smaller accuracy or F1 score drops. For instance, on ResNet models, SQS achieved compression factors of up to 32x with minimal accuracy reduction. Similarly, for BERT-base and large language models like Llama3.2 and Qwen2.5, SQS demonstrated superior compression with competitive performance.
Further studies highlighted the effectiveness of SQS’s individual components. The spike-and-slab distribution proved more efficient in promoting sparsity and preserving model accuracy compared to traditional Gaussian priors. Additionally, using Bayesian averaging during inference consistently yielded better accuracy than a simpler “greedy” decoding approach. A particularly important feature for large language models, which often have weights with extreme values, is the “outlier-aware windowing” strategy. This technique helps preserve these informative large-magnitude weights during quantization, further improving performance.
Also Read:
- Enhancing LLM Efficiency with Fine-grained Low-Rank Compression
- The Locality Dial: Bridging Interpretability and Performance in LLMs
In conclusion, SQS represents a significant advancement in neural network compression. By integrating pruning and quantization into a single, principled Bayesian framework, it offers a powerful tool for deploying complex DNNs on resource-constrained devices without sacrificing performance. This method opens new possibilities for making advanced AI models more accessible and efficient across various applications. You can read the full research paper for more technical details. Read the full paper here.
