TLDR: A new research paper by Defilippis et al. provides a systematic analysis of scaling laws for quadratic and diagonal neural networks in the feature learning regime. By mapping these networks to problems in matrix compressed sensing and LASSO, the authors derive a detailed phase diagram for excess risk, uncovering transitions between scaling regimes and linking them to the spectral properties of learned weights. The study offers a theoretical explanation for empirical observations connecting power-law tails in weight spectra to generalization performance, and demonstrates the surprising robustness of approximate message passing techniques.
Recent advancements in deep learning have been significantly influenced by neural scaling laws, which describe how model performance changes with increased data, model size, and computational resources. However, a deep theoretical understanding of these laws, especially for complex neural networks beyond simple linear models, has remained a considerable challenge.
A new research paper, titled “Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime,” by Leonardo Defilippis and his colleagues, delves into this complex area. The authors present a detailed analysis of scaling laws for two specific types of shallow neural networks: quadratic and diagonal networks. Their work focuses on the ‘feature learning regime,’ where the network actively learns and adapts its internal representations rather than relying on fixed features.
Bridging Neural Networks and Signal Processing
The core innovation of this research lies in establishing a precise connection between these neural network training problems and well-understood tasks in signal processing. Specifically, they show that:
- Diagonal neural networks, when trained with a specific type of regularization (L2 weight decay), can be mathematically mapped to the LASSO problem, which is used for sparse signal recovery.
- Quadratic neural networks, particularly over-parameterized two-layer networks with quadratic activation, can be mapped to matrix compressed sensing, a technique for estimating low-rank matrices from limited measurements.
These equivalences are crucial because they allow the researchers to leverage a rich set of theoretical tools developed for LASSO and compressed sensing, such as approximate message passing (AMP) and its state evolution equations. This bridge provides a powerful framework for making accurate predictions about generalization error and understanding the distribution of network weights.
Uncovering a Universal Phase Diagram
One of the paper’s most significant contributions is the derivation of a comprehensive phase diagram that illustrates the scaling exponents of the ‘excess risk’ – essentially, how much the network’s predictions deviate from the true underlying function. This diagram maps the excess risk as a function of sample complexity (the amount of training data) and weight decay (a regularization technique). The findings reveal a striking universality between diagonal and quadratic networks, including transitions between distinct scaling regimes and plateau behaviors. These theoretical predictions remarkably mirror phenomena widely observed in empirical studies of neural scaling.
The analysis also uncovers a transition from what the authors call ‘benign overfitting’ to ‘harmful overfitting.’ Benign overfitting occurs when a model fits the training data perfectly, including noise, but still generalizes well to new data. Harmful overfitting, conversely, leads to poor generalization. The paper shows that optimal regularization can help avoid harmful overfitting and achieve Bayes-optimal rates, which are the best possible performance given the data.
The Spectrum of Learned Weights and Generalization
Beyond predicting error rates, the research also characterizes the ‘spectral properties’ of the trained network weights. The spectrum refers to the distribution of eigenvalues of the weight matrices, which provides insights into how the network stores and processes information. The learned spectrum reflects a delicate balance between the signal (true features), noise, and regularization, exhibiting patterns directly linked to feature learning.
The authors provide a clear interpretation of how this spectrum relates to generalization performance. They decompose the total error into three components: underfitting (features not learned), overfitting (learned noise), and approximation error (inaccuracies in learned features). Each of these components is directly connected to specific spectral statistics of the weights. This offers a first-principles explanation for empirical observations that link the emergence of power-law tails in the weight spectrum to a network’s ability to generalize effectively.
Robustness of Theoretical Tools
The derivations in the paper rely on approximate message passing (AMP) and its state evolution equations, which are typically guaranteed to be rigorously valid only under specific asymptotic conditions (e.g., fixed ratios of network dimensions). However, the authors extend these equations heuristically beyond their proven settings, covering a wider range of scaling behaviors. Through extensive numerical experiments, they demonstrate that state evolution remains surprisingly accurate across the entire parameter space, suggesting a broader applicability of these theoretical tools than previously established.
Also Read:
- New Research Unveils How Model Complexity Links AI’s Reliability and Learning Ability
- The Sparse Web of Thought: A Network Perspective on LLM Reasoning and Learning
Implications for Model Design
This work provides a comprehensive theoretical and empirical understanding of scaling laws for feature learning in simplified network models. It offers valuable insights into why certain scaling behaviors are observed and how regularization strategies impact generalization. The findings also suggest that simple pruning methods, which involve setting smaller eigenvalues of the learned weights to zero, can achieve optimal error rates without the need for complex regularization tuning.
For a more in-depth understanding, the full research paper can be accessed here.
