TLDR: GLASS introduces a novel training-free method to accelerate Large Language Models (LLMs) by dynamically pruning their Feed-Forward Network (FFN) units. It combines prompt-specific ‘local’ activation patterns with precomputed ‘global’ model-intrinsic neuron importance. This approach significantly enhances performance, especially for tasks with short prompts and long generated responses, outperforming existing training-free methods without adding inference overhead, making LLMs more efficient for edge device deployment.
Large Language Models (LLMs) have become incredibly powerful, driving advancements across many natural language applications. However, their immense size and computational demands make deploying them on everyday devices, often called ‘edge hardware,’ a significant challenge. These models require substantial memory and processing power, leading to high inference latency and memory footprint, which can hinder their practical use on consumer-grade hardware.
One common strategy to make LLMs more efficient is ‘structured pruning,’ especially within their Feed-Forward Network (FFN) blocks. This technique aims to reduce inference time and memory usage by removing less important weights or neurons. Traditional pruning methods, whether static (fixed sparsity patterns) or predictor-based (using auxiliary models to anticipate active neurons), often fall short. Static pruning can compromise quality, while predictor-based methods introduce extra training and runtime overhead. Recent ‘zero-shot’ methods, which rely on statistics from a single prompt, struggle with short prompts or long generation tasks because the activation patterns can drift over time.
Introducing GLASS: A Smart Approach to LLM Acceleration
A new approach called A/I-GLASS (Activation- and Impact-based Global-Local neural importance Aggregation for FFN SparSification) offers a compelling solution. GLASS introduces two training-free methods that dynamically select FFN units. Unlike previous methods, GLASS combines ‘local’ neuron statistics, derived from the current prompt, with ‘global’ model-intrinsic neuron statistics. This aggregation helps to more accurately identify the most critical neurons, even in challenging scenarios like short prompts followed by long generated responses.
The core idea behind GLASS is to leverage both immediate context and the model’s inherent knowledge. The ‘local’ information captures what’s important for the current input, while the ‘global’ information provides a broader, precomputed understanding of neuron importance across diverse data. By combining these two perspectives, GLASS ensures that the model retains neurons essential for understanding the prompt while also incorporating general model knowledge to prevent overfitting to a single prompt.
There are two main variants: A-GLASS, which uses activation magnitudes, and I-GLASS, which uses neuron impact on the model’s loss. These global importance profiles are computed only once per model and can be cached, adding negligible storage cost. To obtain these global statistics, the researchers propose ‘Null Prompt Stimulation’ (NPS), where the model generates text from a null prompt (like an empty string) to reflect its intrinsic token distribution, avoiding biases from external datasets.
Also Read:
- QuickMerge++: Streamlining Generative AI with Adaptive Token Compression
- Unpacking Pruning Strategies: A Deep Dive into One-Shot, Iterative, and Hybrid Model Compression
Performance and Benefits
The empirical results for GLASS are impressive. Across multiple LLMs and benchmarks, GLASS significantly outperforms prior training-free methods, particularly in long-form generation scenarios. For instance, on the Gemma 2 9B model, I-GLASS achieved a 12.91% reduction in Perplexity (PPL) and a 16.48% reduction in Kullback-Leibler Divergence (KLD) compared to GRIFFIN, a strong baseline. For larger models like Gemma 2 27B, I-GLASS reduced PPL by 12.48% and KLD by 22.80%.
A particularly notable improvement was observed on the Gemma 3n family of models, which are specifically designed for efficient on-device deployment. On Gemma 3n E4B, A-GLASS and I-GLASS showed 31.30% and 26.43% improvements in PPL, and 25.26% and 22.25% improvements in KLD, respectively. These substantial gains highlight GLASS’s effectiveness in scenarios requiring aggressive sparsity, making it highly suitable for edge devices.
Crucially, GLASS achieves these improvements without relying on auxiliary predictors or adding any inference overhead, which is a major advantage for latency-sensitive deployments. This means faster and more efficient LLM inference on devices without compromising output quality.
This research demonstrates that integrating model-intrinsic global knowledge can dramatically enhance prompt-driven sparsification, leading to reliable speed-ups for the short-prompt, long-generation workloads common on edge devices. For more technical details, you can refer to the full research paper: GLASS: Test-Time Acceleration for LLMs via Global-Local Neural Importance Aggregation.
