TLDR: A new research paper introduces ‘mixup-class prompt,’ a novel text prompting strategy for Data-Free Quantization (DFQ). This method generates diverse and robust synthetic images by fusing multiple class labels in text prompts, effectively addressing issues like polysemy and limited data diversity. It significantly enhances the generalization and optimization stability of quantized AI models, achieving state-of-the-art accuracy in low-bit quantization settings for both CNNs and Vision Transformers.
In the world of artificial intelligence, making powerful models run efficiently on devices with limited resources is a major challenge. One popular technique to achieve this is called quantization, which reduces the size and computational cost of deep learning models by converting their high-precision data into lower-bit representations. Among various quantization methods, Post-Training Quantization (PTQ) is particularly appealing because it doesn’t require retraining the model, saving significant time and resources.
However, PTQ often relies on a small amount of calibration data, which can lead to models that don’t generalize well to new, unseen data. This is especially problematic in situations where access to real training data is restricted, perhaps due to privacy concerns. To overcome this, Data-Free Quantization (DFQ) has emerged. DFQ eliminates the need for real calibration data by generating synthetic images using advanced generative models like Latent Diffusion Models (LDMs).
While generative models can create synthetic data, a key issue arises: the quality and diversity of these generated images. Current methods often use simple, single-class text prompts (e.g., just the word “kite”) to generate images. This can lead to problems like polysemy, where a word has multiple meanings. For instance, “kite” can refer to a bird or a toy. If the generative model is biased towards creating images of a toy kite when the intended class is a bird, the synthetic data won’t accurately represent the real data distribution. This mismatch can severely degrade the performance of the quantized model, especially since PTQ uses a very small calibration set.
To address this, researchers have proposed a novel strategy called “mixup-class prompt”. Instead of using a single class label, this method fuses multiple class labels at the text prompt level (e.g., “a realistic photo of a kite and a vulture”). This innovative approach encourages the LDM to generate synthetic images that contain characteristics of multiple classes, leading to more diverse and robust data. By blending ambiguous objects into broader contexts, mixup-class prompting effectively mitigates the polysemy issue and enhances the generalization capability of the quantized model.
The effectiveness of mixup-class prompting is supported by quantitative analysis. The research shows that this method leads to a lower empirical gradient norm during the quantization process, which theoretically indicates a smaller generalization gap and improved optimization stability. In simpler terms, the model is less likely to overfit to the synthetic data and performs better on real-world examples.
Extensive experiments were conducted on various model architectures, including Convolutional Neural Networks (CNNs) like ResNet and MobileNetV2, and Vision Transformers (ViTs) like ViT, Swin, and DeiT. The results consistently demonstrate that mixup-class prompting outperforms existing state-of-the-art DFQ methods, such as GenQ. It achieves superior accuracy across different bit-width settings, pushing the performance boundary, especially in extremely low-bit scenarios like 2-bit weight and 4-bit activation (W2A4) quantization, where it sets new accuracy benchmarks.
Further studies confirmed the robustness of this approach. It was found to be more effective than traditional image augmentation techniques and other prompt engineering strategies. Moreover, the benefits of mixup-class prompting hold true across different versions of Stable Diffusion models, indicating its broad applicability. Interestingly, random pairing of class labels for the mixup prompt yielded the best results, suggesting that diversity is more crucial than semantic similarity between the mixed classes. The research also found that combining two classes in the prompt is optimal, as adding more classes can dilute the semantic clarity and reduce effectiveness.
Also Read:
- DiCap: Enhancing AI Prompt Learning with Causal Insights and Diffusion Models
- Achieving Unified Styles in AI-Generated Multi-Object Images
This work represents a significant step forward in Data-Free Quantization, offering a simple yet powerful method to generate high-quality synthetic data for efficient deep learning model deployment. For more technical details, you can refer to the full research paper: Enhancing Generalization in Data-free Quantization via Mixup-class Prompting.
