TLDR: This research systematically compares one-shot and iterative pruning strategies for neural network compression. It finds that one-shot pruning is better for lower compression ratios, while iterative pruning excels at higher ratios and for transformer models. The study introduces a more effective geometric pruning scheduler and advocates for patience-based fine-tuning. A novel hybrid pruning approach, combining the strengths of both methods, is also proposed, demonstrating improved performance across various scenarios. The findings provide practical guidelines for selecting optimal pruning strategies based on specific goals and computational resources.
Neural networks, the backbone of modern artificial intelligence, are becoming increasingly complex and large. While this often leads to superior performance, it also makes them computationally expensive and resource-intensive, especially for devices with limited power like mobile phones or embedded systems. This is where ‘pruning’ comes in – a crucial technique for compressing these networks by removing redundant parts without significantly compromising their performance.
Traditionally, pruning has been approached in two main ways: one-shot pruning and iterative pruning. One-shot pruning involves a single, comprehensive pass of training and pruning, where a large portion of the network’s weights are removed at once. In contrast, iterative pruning performs pruning over multiple cycles, gradually refining the network structure by repeatedly pruning small amounts and then retraining the model. While iterative pruning has been more widely adopted, its superiority has often been assumed rather than rigorously tested.
A recent study, titled One Shot vs. Iterative: Rethinking Pruning Strategies for Model Compression, conducted by Mikołaj Janusz, Tomasz Wojnar, Yawei Li, Luca Benini, and Kamil Adamczewski, provides one of the first systematic and comprehensive comparisons of these two methods. Their research offers rigorous definitions, benchmarks both structured and unstructured pruning settings, and applies different pruning criteria and modalities to understand their effectiveness.
The study reveals that each method has distinct advantages depending on the scenario. One-shot pruning proves more effective at lower pruning ratios, meaning when you want to remove a smaller percentage of the network. On the other hand, iterative pruning performs better at higher pruning ratios, where a significant portion of the network is being removed. This finding is particularly relevant for practitioners who need to choose a pruning strategy tailored to their specific goals and computational constraints.
Key Innovations and Findings
One significant contribution of this research is the introduction of a geometric pruning ratio scheduler for iterative pruning. Unlike the constant scheduler, which prunes a fixed percentage of weights across the entire network in each step, the geometric scheduler prunes a fixed percentage of the *remaining* weights. This means progressively fewer weights are removed as pruning advances, and experiments show that this geometric approach generally outperforms the constant scheduler.
The researchers also advocate for ‘patience-based pruning,’ which uses early stopping to determine the optimal fine-tuning duration. Instead of arbitrarily fixing retraining epochs, this adaptive approach trains the model until its performance (e.g., validation accuracy) no longer improves over a set number of epochs. This ensures that the model is retrained just enough, avoiding both insufficient and excessive training, which can waste computational resources or even degrade performance.
Also Read:
- Unlocking Efficient Data Learning with Compressive Meta-Learning
- Piecewise-Affine Regularization: A Framework for Quantized Machine Learning Models
Hybrid Approach and Practical Implications
Building on their findings, the study introduces a novel ‘hybrid few-shot pruning’ regime. This approach combines the strengths of both one-shot and iterative methods. It starts by removing a large portion of weights in an initial one-shot-like step, followed by a more fine-grained, geometric pruning strategy for the remaining weights. This hybrid method demonstrated superior performance across nearly all pruning rates, especially enhancing results at lower pruning rates.
The empirical evaluation covered a wide range of settings, including vision datasets like CIFAR-10, CIFAR-100, and Imagenet1K, and the language dataset TinyStories. They tested on various model architectures, including convolutional neural networks (ResNet, EfficientNet) and transformers (Visual Transformer, TinyStories-33M). For natural language processing tasks, iterative pruning showed a notable advantage at higher compression rates, and interestingly, perplexity often decreased as pruning progressed, suggesting that large language models can benefit significantly from pruning redundant parameters.
From a computational perspective, the study found that one-shot pruning is more efficient for pruning rates up to 80% across various computational budgets. However, at higher pruning rates, iterative pruning becomes the preferred method. The choice of pruning criterion (e.g., magnitude-based, Taylor Expansion, Hessian-based) also influences the optimal strategy, with second-derivative methods performing better in one-shot scenarios at lower pruning ratios, and magnitude-based pruning being advantageous for iterative approaches at higher ratios due to its lower computational cost.
In conclusion, this research provides valuable guidelines for practitioners navigating the complex landscape of neural network compression. It emphasizes that selecting the most suitable pruning strategy, along with key hyperparameters like retraining length and step size, should be carefully tailored to specific performance objectives and computational constraints. The insights from this study pave the way for more informed and effective pruning practices in the future.
