TLDR: Researchers introduced Orthogonal Natural Gradient Descent (ONG), combining natural gradients (which account for parameter space geometry) with orthogonal gradient descent (which prevents forgetting by projecting new task gradients). While theoretically sound and using an efficient Fisher matrix approximation (EKFAC), initial experiments on Permuted and Rotated MNIST showed ONG performed worse than standard orthogonal gradient descent, highlighting challenges in combining these methods and suggesting future research directions in manifold learning and diverse datasets.
In the rapidly evolving field of artificial intelligence, a significant challenge known as “continual learning” seeks to enable models to learn new tasks without forgetting previously acquired knowledge. Imagine a student who learns algebra, then geometry, and then calculus, but completely forgets algebra once they start geometry. This is analogous to “catastrophic forgetting” in neural networks, where training on new tasks can erase performance on older ones.
Researchers Yajat Yadav, Jathin Korrapati, and Patrick Mendoza from UC Berkeley have introduced a novel approach called Orthogonal Natural Gradient Descent (ONG) to tackle this very problem. Their work, detailed in the paper ONG: Orthogonal Natural Gradient Descent, aims to combine the strengths of two powerful optimization techniques: Orthogonal Gradient Descent and Natural Gradient Descent.
Understanding the Foundations: OGD and Natural Gradients
Orthogonal Gradient Descent (OGD) is a method designed to prevent catastrophic forgetting. It works by ensuring that when a model learns a new task, its updates are “orthogonal” to the knowledge gained from previous tasks. This means that the new learning direction doesn’t interfere with the directions established by prior learning, effectively preserving performance on old tasks.
On the other hand, Natural Gradient Descent (NGD) takes a different approach to optimization. Traditional gradient descent treats the parameter space of a neural network as a flat, Euclidean space. However, the true “information geometry” of this space, especially for probabilistic models, is curved. NGD accounts for this curvature by preconditioning the gradient with the inverse Fisher Information Matrix. This ensures that the optimization steps follow the steepest descent direction in this curved space, leading to faster convergence and updates that are invariant to how the model’s parameters are represented.
Introducing ONG: A Hybrid Approach
The core idea behind ONG is to merge these two powerful concepts. ONG preconditions each new task gradient using an efficient approximation of the inverse Fisher Information Matrix. This step ensures that the gradient update respects the underlying information-geometric structure of the model’s parameter space. After this “natural preconditioning,” ONG then projects these natural gradients onto the orthogonal complement of prior task gradients. This projection step is crucial for preserving performance on previously learned tasks, much like in OGD.
To make the computation feasible for deep neural networks, ONG utilizes the Eigenvalue-Corrected Kronecker Factorization (EKFAC) to approximate the Fisher Information Matrix. This approximation significantly reduces the computational burden while still providing a close estimate of the true Fisher matrix.
Theoretical Insights and Experimental Results
The researchers provide theoretical justification for ONG, demonstrating that the preconditioned and projected gradient still constitutes a valid descent direction with respect to the Fisher metric. This ensures that the algorithm continues to optimize the loss function effectively.
However, the experimental results on standard continual learning benchmarks, Permuted MNIST and Rotated MNIST, presented an interesting outcome. Surprisingly, ONG and its variant, ONG+, performed worse in terms of average forgetting and average validation accuracy compared to their Euclidean counterparts, OGD and OGD+. This suggests that while both natural gradients and orthogonal projections are individually powerful, their direct combination presents unique challenges, particularly in the context of these specific datasets.
Also Read:
- Unpacking Task Arithmetic: How Early Training Dynamics Drive Model Merging
- Geometric Optimization Enhances LoRA for Large Language Models
Future Directions
Despite the initial experimental results, the authors believe that ONG opens up exciting avenues for future research. They propose incorporating insights from Manifold Learning and Information Geometry, such as the concept of “parallel transports,” to better align natural gradients in a common tangent space for projections. Further exploration into the parameterization-invariance property of natural gradients and testing ONG on more diverse and uncorrelated datasets are also highlighted as important next steps. The hypothesis is that the benefits of natural gradient-based methods might become more pronounced in more challenging and varied learning scenarios.
