TLDR: PermLLM is a new framework that improves the accuracy of N:M sparse Large Language Models by learning the optimal way to reorder internal data channels. Unlike traditional methods that use fixed rules, PermLLM directly optimizes channel permutations to minimize pruning errors, using a block-wise strategy for efficiency and achieving superior performance across various LLMs.
Large Language Models (LLMs) have become incredibly powerful, but their massive size makes them challenging to deploy efficiently. To tackle this, researchers often use model compression techniques like pruning, which involves removing less important parts of the model. One specific and efficient type of pruning is N:M sparsity, where a fixed number of weights (N) are kept out of every M consecutive weights. This pattern is particularly beneficial because it’s supported by specialized hardware like NVIDIA GPUs, leading to faster computations.
A key technique to improve the accuracy of N:M sparse models is “channel permutation.” Imagine the model’s internal data channels as a deck of cards; channel permutation is like reordering these cards to ensure that the most important information is retained after pruning. Traditionally, this reordering relied on predefined rules or “handcrafted quality metrics” to decide which channels are most important. However, these metrics don’t always perfectly reflect how pruning affects the model’s overall performance, sometimes even leading to worse results.
This is where PermLLM comes in. PermLLM is a new framework that introduces “learnable channel permutation” (LCP). Instead of relying on fixed rules, PermLLM *learns* the best way to reorder channels by directly minimizing the errors caused by pruning. It does this by transforming the complex, discrete problem of reordering into a continuous, differentiable one using a technique called Sinkhorn normalization. This allows the model to use standard optimization methods to find the ideal channel arrangement.
To make this learning process efficient, especially for large LLMs, PermLLM uses a “block-wise” strategy. Instead of trying to reorder all channels across the entire model at once (which would be computationally very expensive), it divides the channels into smaller blocks and learns permutations within these blocks. This significantly reduces the number of parameters that need to be learned and speeds up the process. For example, a block size of 64 was found to offer a good balance between performance and efficiency.
PermLLM is designed to work seamlessly with existing one-shot pruning methods like Wanda and RIA. It acts as a “plugin” that optimizes channel permutations to reduce pruning-induced errors, ultimately leading to better-performing sparse models. The framework directly minimizes the difference between the output of the original, full-sized model and the pruned, sparse model using a cosine similarity loss.
The researchers conducted extensive experiments on various LLMs, including the LLaMA series, Qwen, and OPT models. The results showed that PermLLM consistently achieved superior performance in optimizing N:M sparse models compared to traditional methods. For instance, on the Wikitext2 language modeling task, PermLLM significantly improved perplexity (lower is better) across different models. It also boosted zero-shot performance on tasks like HellaSwag and ARC.
Furthermore, the team developed a specialized CUDA kernel to accelerate the channel permutation operation, achieving an impressive 84x speedup compared to a standard Pytorch implementation. This makes the learnable channel permutation practical for real-world deployment, with the overall acceleration across linear layers being approximately 1.67x.
Also Read:
- Optimizing LLM Deployment: Introducing AnyBCQ for Flexible and Efficient Multi-Precision Inference
- Enhancing LLM Efficiency with Fine-grained Low-Rank Compression
PermLLM represents a significant step forward in making large language models more efficient without sacrificing accuracy. By learning how to best arrange channels, it helps sparse models retain more of their original performance, paving the way for broader and more efficient deployment of these powerful AI tools. You can find more details about this innovative work in the full research paper available at this link.
