TLDR: A new deep learning optimizer, SoftSignSGD (S3), is introduced to overcome Adam’s limitations, particularly loss spikes. S3 features a generalized sign-like update, unified momentum coefficients for stability, and Nesterov’s accelerated gradient for faster convergence. Experiments show S3 trains deep neural networks more efficiently, achieves better performance, and significantly reduces loss spikes compared to existing optimizers like AdamW.
Optimizers are fundamental to training deep neural networks (DNNs), with Adam being a widely adopted choice, especially for large language models like GPT-3 and vision models such as CLIP. While Adam has achieved significant practical success, the precise reasons for its effectiveness and its limitations have not been fully understood.
A recent study delves into Adam’s behavior, revealing that its success largely stems from its similarity to SignSGD, which is effective at handling large fluctuations in gradients. However, the research also points out a critical vulnerability: Adam’s uncontrolled update scaling can lead to destabilizing ‘loss spikes’ during training. These spikes are a common and problematic phenomenon, particularly in the training of large models, often requiring practitioners to resort to costly workarounds like restarting training from previous checkpoints.
To address these challenges, a novel optimizer called SoftSignSGD (S3) has been proposed. S3 introduces three key innovations designed to enhance Adam’s advantages while mitigating its drawbacks:
Generalized Sign-Like Update
S3 moves beyond the conventional second-order momentum (variance) preconditioning found in Adam. Instead, it employs a flexible p-th order momentum (where p is greater than or equal to 1) in its denominator. This design allows for more stable training, even when using aggressive learning rates, and contributes to enhanced performance.
Loss Spike Minimization
A significant feature of S3 is its ability to minimize the occurrence of loss spikes. It achieves this by using unified exponential moving average coefficients for both the numerator and denominator momenta. This inherent design bounds updates to a range of [-1, 1], which directly limits the maximum update magnitude and prevents the disproportionately large updates that cause instability in Adam. This also simplifies hyperparameter tuning by reducing the number of parameters and eliminating the need for bias correction and gradient clipping.
Also Read:
- A New View on Adam’s Convergence: Simpler Proofs and Deeper Insights
- NeVe: A Novel Approach to Efficient Neural Network Training Without Validation Data
Nesterov’s Accelerated Gradient (NAG) Integration
S3 incorporates an equivalent Nesterov’s accelerated gradient (NAG) module. This integration accelerates the convergence of the training process without incurring additional memory overhead, making it particularly suitable for large models where memory efficiency is crucial.
Theoretically, S3 has been proven to achieve an optimal convergence rate for general nonconvex stochastic optimization under weak assumptions. Extensive experiments across various vision and language tasks demonstrate that S3 not only converges more rapidly and improves overall performance but also rarely experiences loss spikes, even when trained with learning rates up to 10 times larger than those used with Adam. In fact, S3 can deliver comparable or superior performance to AdamW while requiring half the training steps, showcasing its efficiency and effectiveness in real-world applications.
This research provides a deeper understanding of Adam’s underlying mechanisms and offers a robust solution to the problem of loss spikes in large model training, paving the way for more stable and efficient deep learning. You can read the full research paper here.
