TLDR: This research paper investigates the Stop Gradient (SG) and Exponential Moving Average (EMA) procedures, commonly used in non-contrastive self-supervised learning to prevent representation collapse. From an optimization viewpoint, the study reveals that these methods generally do not optimize the original objective function, nor any other smooth function, yet they effectively avoid collapse. From a dynamical systems perspective, particularly in the linear case, the paper demonstrates that the original objective without SG/EMA always leads to collapse, while SG/EMA procedures lead to asymptotically stable equilibria, explaining their practical success in learning meaningful representations.
Self-supervised learning (SSL) has emerged as a powerful alternative to traditional supervised methods, allowing models to learn valuable representations from data without requiring expensive human annotations. This approach has found success in various fields, including natural language processing, image analysis, and video understanding. However, a significant challenge in non-contrastive SSL methods is the risk of ‘representational collapse,’ where the model learns a trivial, constant embedding for all data points, rendering the learned representations useless.
To combat this, two iterative procedures, the Stop Gradient (SG) and Exponential Moving Average (EMA), are commonly employed. These techniques have shown excellent practical performance in preventing collapse and achieving strong results in downstream applications. A recent research paper, titled “Dual Perspectives on Non-Contrastive Self-Supervised Learning,” delves into these procedures from two distinct viewpoints: optimization and dynamical systems, offering a deeper theoretical understanding of their behavior.
The Optimization Puzzle
The paper, authored by Jean Ponce, Basile Terver, Martial Hebert, and Michael Arbel, first investigates SG and EMA from an optimization perspective. A key finding is that, in general, neither the SG nor the EMA algorithm actually minimizes the original objective function that they are derived from. Furthermore, the authors prove a conjecture that these algorithms do not optimize any other smooth function either. This is a crucial insight, as it challenges the conventional understanding of these methods as direct optimization procedures.
Despite not optimizing a clear objective, the research demonstrates that when these procedures converge, they generally succeed in avoiding representational collapse. Empirical experiments using real-world video classification data (Kinetics710 and SSv2 benchmarks) further illustrate this point. The experiments show that while the objective function values do not steadily decrease and the parameters do not fully converge in the traditional sense, the classification accuracy on downstream tasks initially increases, confirming that the algorithms learn something meaningful.
A Dynamical Systems View
The paper then shifts to a dynamical systems perspective, particularly focusing on the linear case where the encoder and predictor are linear operators. Here, the findings are even more striking: minimizing the original objective function without the use of SG or EMA *always* leads to collapse. This highlights the critical role these procedures play in preventing degenerate solutions.
Conversely, when viewed as dynamical systems, the SG and EMA algorithms exhibit stable behavior. The researchers explicitly characterize the ‘equilibria’ (stable states) of these systems as algebraic varieties in their parameter space. Crucially, they prove that these equilibria are, in general, asymptotically stable. This means that if the training process starts close to one of these stable points, it is guaranteed to converge to and remain at that point, effectively avoiding collapse.
Experiments with synthetic data, where the input space is simplified, further support these theoretical findings. Simulations show that the algorithms consistently converge, mostly to non-trivial limit points on the identified stable varieties, and rarely to the degenerate origin. This provides strong evidence for the stability properties of SG and EMA.
Also Read:
- Unlocking AI’s Potential: Deep Learning’s New Foundation in Differential Equations
- Decoding Quantized Model Training: How Bit Width and Range Shape Learning Dynamics
Conclusion: Effective, Even if Unconventional
In summary, the research paper concludes that while the SG and EMA algorithms are not conventional optimization procedures that minimize a well-defined objective function, they are highly effective at preventing representational collapse. Their success can be attributed to their properties as dynamical systems, which guide the training process towards asymptotically stable, non-trivial equilibria. This deeper understanding helps explain why these methods, despite their unconventional nature, yield excellent empirical results in self-supervised learning. For more details, you can read the full paper here.
