TLDR: New research reveals that using large learning rates during machine learning model training simultaneously enhances a model’s robustness against misleading “spurious correlations” in data and improves its compressibility, making it more resource-efficient. This dual benefit is consistent across various models and datasets, and the mechanism involves large learning rates amplifying errors from challenging data points, thereby forcing the model to focus on essential, core features.
In the rapidly evolving world of artificial intelligence, two critical challenges for modern machine learning models are ensuring they are robust enough to handle new, unseen situations and that they are resource-efficient, capable of running on smaller hardware with less computational power. Traditionally, achieving both simultaneously has been a significant hurdle. However, new research sheds light on a surprising facilitator for these properties: large learning rates.
A recent paper titled “Large Learning Rates Simultaneously Achieve Robustness to Spurious Correlations and Compressibility” by Melih Barsbey, Lucas Prieto, Stefanos Zafeiriou, and Tolga Birdal from Imperial College London, positions high learning rates as a pivotal factor in addressing these dual challenges. The study demonstrates that using large learning rates during the training of deep neural networks can simultaneously lead to models that are more robust against misleading patterns in data and are more compressible, meaning they can be made smaller and more efficient without losing performance.
Understanding Spurious Correlations
To grasp the significance of this finding, it’s important to understand what spurious correlations are. Imagine training a model to identify cows, but in your training data, most cows are pictured on green grass. If the model learns to associate “green background” with “cow,” it might misclassify a cow on a sandy beach as a camel because camels are often seen in deserts. This is a spurious correlation: a relationship in the training data that doesn’t hold true in the real world or in different environments. Models that rely on these shortcuts perform poorly when faced with out-of-distribution (OOD) data.
The Impact of Learning Rates
The learning rate (LR), or step size, is a crucial hyperparameter in gradient-based training. It determines how much the model’s weights are adjusted in response to the estimated error each time it learns from data. The researchers found that large learning rates not only improve robustness to these spurious correlations but also enhance network compressibility. This means the models can be effectively pruned or simplified, making them more resource-efficient without sacrificing their ability to generalize.
Beyond robustness and compressibility, large learning rates were also observed to produce other desirable representation properties within the neural network. These include better utilization of invariant features (the true, core characteristics of an object, like the shape of a cow, rather than its background), improved class separation (making it easier for the model to distinguish between different categories), and activation sparsity (where fewer neurons are active, leading to more efficient computation).
Also Read:
- Unpacking Neural Network Behavior: Capacity, Sparsity, and Resilience in Smaller Models
- Neural Networks’ Dynamic Adaptation: How Task Shifts Reactivate Learning
Generalization and Mechanism
The findings were consistent across a wide range of architectures, datasets (including semi-synthetic and naturalistic ones like CelebA and Waterbirds), and optimizers. This suggests that the positive effects of large learning rates are not limited to specific setups but are a general phenomenon. Interestingly, the study also provides strong evidence that the previously documented success of large learning rates in standard classification tasks (where spurious correlations might not be immediately obvious) is likely due to their effect on addressing hidden or rare spurious correlations present in the training dataset.
So, how do large learning rates achieve this? The paper proposes a fascinating mechanism: during training, simple, spurious features are often learned earlier than more complex, core features. At this stage, large learning rates lead to “confident mispredictions” of samples that conflict with the learned bias (e.g., a cow on sand). Because of the nature of the loss function used (cross-entropy), these confident mispredictions generate very strong gradients. These strong gradients effectively reweight the dataset, implicitly emphasizing the challenging, bias-conflicting examples. This pressure discourages the model from relying on spurious features and forces it to learn the more robust, core features instead. This process can be thought of as a recurring “lottery ticket” scenario, where spurious subnetworks are effectively reset by large updates when they over-rely on misleading features.
This research highlights the profound implicit bias of learning rates in shaping the behavior of deep neural networks. By demonstrating that large learning rates can simultaneously achieve robustness to spurious correlations and network compressibility, this work offers crucial insights for designing the next generation of safe, trustworthy, and resource-efficient machine learning models. For more details, you can refer to the full research paper available at arXiv:2507.17748.
