TLDR: HierarchicalPrune is a new compression framework for large text-to-image diffusion models (DMs) that significantly reduces their memory footprint and improves speed while preserving image quality. It achieves this by recognizing and leveraging the “hierarchical” importance of different parts of the DM, strategically pruning less essential components and carefully distilling knowledge. The method combines Hierarchical Position Pruning, Positional Weight Preservation, and Sensitivity-Guided Distillation, resulting in up to 80.4% memory reduction and 38% latency reduction with minimal quality loss, making billion-scale DMs viable for resource-constrained devices.
Large-scale text-to-image diffusion models (DMs) have revolutionized image generation, creating stunning visuals from text prompts. However, their immense size, often reaching 8-11 billion parameters, makes them challenging to run on everyday devices like smartphones or consumer-grade graphics cards. This limitation restricts their widespread use and accessibility.
A new research paper, “HierarchicalPrune: Position-Aware Compression for Large-Scale Diffusion Models”, introduces an innovative compression framework designed to tackle this very problem. Developed by Young D. Kwon, Rui Li, Sijia Li, Da Li, Sourav Bhattacharya, and Stylianos I. Venieris, HierarchicalPrune aims to make these powerful models more accessible by significantly reducing their memory footprint and improving inference speed, all while maintaining high image quality.
Understanding the Core Idea: A Dual Hierarchy
The foundation of HierarchicalPrune lies in a crucial observation about how diffusion models work. The researchers discovered that different parts, or “blocks,” within these models have distinct roles. Early blocks are responsible for establishing the fundamental semantic structure of an image (like the main objects and their layout), while later blocks handle the finer details and textures. This is referred to as an “inter-block hierarchy.” Additionally, within each block, individual subcomponents also have varying levels of importance, forming an “intra-block hierarchy.”
Traditional compression methods often treat all parts of the model uniformly, which can lead to significant quality degradation when trying to achieve high compression rates. HierarchicalPrune, however, leverages this newly identified dual hierarchy to apply compression more intelligently.
The Three Pillars of HierarchicalPrune
HierarchicalPrune combines three synergistic techniques:
1. Hierarchical Position Pruning (HPP): This technique identifies and removes less essential blocks, primarily focusing on later blocks in the model’s architecture. Since early blocks are critical for semantic structure, HPP strategically preserves them, ensuring the core image composition remains intact while pruning deeper layers responsible for refinements.
2. Positional Weight Preservation (PWP): During the model’s refinement process (known as distillation), PWP “freezes” the weights of the non-pruned and earlier parts of the model. This protection ensures that the foundational elements, crucial for image formation, are not inadvertently altered, allowing later, less critical blocks to be fine-tuned.
3. Sensitivity-Guided Distillation (SGDistill): For more aggressive compression, the researchers found that even important blocks can be highly sensitive to changes during distillation. SGDistill employs a counterintuitive approach: it assigns minimal or zero update weights to these highly sensitive, important blocks, concentrating updates on less sensitive components. This prevents detrimental quality drops that would otherwise occur when aggressively compressing the model.
As a final step, HierarchicalPrune can optionally combine these techniques with INT4 weight quantization, which further reduces the model’s size by representing weights with fewer bits.
Also Read:
- Enhancing Web Agent Performance with Efficient DOM Processing
- Sculptor: Giving Language Models the Power to Manage Their Own Thoughts
Impressive Results and User Validation
The effectiveness of HierarchicalPrune was rigorously tested on state-of-the-art diffusion models like SD3.5 Large Turbo (8 billion parameters) and FLUX.1-Schnell (12 billion parameters). The results are compelling:
- Memory Footprint Reduction: HierarchicalPrune achieved a remarkable 77.5-80.4% memory reduction. For instance, the SD3.5 Large Turbo model’s memory usage dropped from 15.8 GB to just 3.2 GB, making it suitable for on-device inference.
- Latency Reduction: The framework also delivered a significant speedup, with 27.9-38.0% reduction in inference latency.
- Quality Preservation: Crucially, these reductions came with only a minimal drop in image quality. Quantitative metrics like GenEval and HPSv2 showed a drop of just 2.6% and 7% respectively, compared to the original model.
- User Study Validation: An extensive user study involving 85 participants further confirmed the perceptual quality. HierarchicalPrune maintained image quality comparable to the original model, significantly outperforming prior compression methods which showed substantial degradation. The user study revealed only a 4.8-5.3% degradation in user-perceived quality, in stark contrast to 11.1-52.2% degradation seen in prior works.
This research marks a significant step towards democratizing access to high-quality text-to-image generation, enabling powerful diffusion models to run efficiently on a wider range of devices, from cloud servers to consumer-grade GPUs.
