TLDR: A research paper explains that low-precision training slows down deep learning convergence because gradient quantization reduces the effective stepsize of SGD and introduces noise, leading to a slower training rate and a higher final error floor, even though convergence still occurs.
In the rapidly evolving world of deep learning, the sheer size and complexity of models demand significant computational and memory resources. To tackle this, researchers have turned to low-precision training, using formats like FP16, FP8, and FP4 instead of the standard FP32. While these methods effectively cut down on resource usage and speed up training, they often come with a trade-off: reduced accuracy and potential numerical instability.
A recent research paper, “SGD Convergence under Stepsize Shrinkage in Low-Precision Training,” by Vincent-Daniel Yun and Juyoung Yun from the University of Southern California, delves into a critical aspect of this challenge. The authors investigate how the process of quantizing gradients—a key step in low-precision training—introduces two main issues: a reduction in gradient magnitude (shrinkage) and the addition of random noise.
The core idea presented in the paper is that this gradient shrinkage effectively reduces the “stepsize” used in Stochastic Gradient Descent (SGD), the optimization algorithm widely used to train deep learning models. Imagine SGD as taking steps towards the optimal solution; if each step is systematically made smaller due to shrinkage, the journey will naturally take longer. The paper models this by showing that the nominal stepsize (µk) is replaced by an “effective stepsize” (µkqk), where qk is the shrinkage factor. When qk is less than 1, convergence slows down.
The researchers provide a rigorous theoretical analysis, building upon standard SGD convergence frameworks. They prove that even with gradient shrinkage and quantization noise, low-precision SGD still converges. However, this convergence occurs at a reduced rate, directly influenced by the minimum shrinkage factor (qmin). Furthermore, the quantization noise contributes to an increased “asymptotic error floor,” meaning the model might not reach the same level of accuracy as its full-precision counterpart, even after extensive training.
The paper highlights that this slowdown is a direct consequence of the effective stepsize being smaller. For both fixed and diminishing stepsize schedules, the theoretical bounds derived in the paper explicitly show how the reduced effective stepsize impacts the rate of convergence and the final error. This provides a clear theoretical explanation for why low-precision networks often train slower and achieve slightly lower performance compared to full-precision ones.
This work is crucial because it offers a deeper understanding of the underlying mechanisms that affect low-precision training. By quantifying the impact of gradient shrinkage, the findings can guide future strategies for designing more effective stepsize schedules and optimization techniques specifically tailored for low-precision environments. It complements existing research on low-precision training by focusing on the often-overlooked aspect of gradient shrinkage’s direct influence on the effective stepsize.
Also Read:
- Unpacking Energy Use in Parallel Neural Network Training
- Smart LLM Adaptation: DP-LLM Adjusts Precision on the Fly
For more in-depth details, you can read the full research paper available at arXiv:2508.07142.
