TLDR: This research explores how symmetry and network complexity affect the ability of neural networks (MLPs and GNNs) to learn physical transformations, using the Central Limit Theorem as a test case. It finds that a delicate balance is needed: symmetry constraints improve generalization only when they align with the task’s requirements, while overly constrained or excessively flexible models tend to perform poorly. The study also extends a framework for tracking statistical properties through network layers, offering insights into their internal information processing.
Deep learning models have achieved remarkable success in various fields, from predicting molecular structures to solving complex physical equations. A key aspect of their power lies in their ability to learn intricate features from structured data through multiple layers of representation. This research paper, titled “Symmetry and Generalisation in Neural Approximations of Renormalisation Transformations,” delves into a fundamental question: how do physical symmetries and the inherent expressiveness of these models influence their ability to generalize, especially when learning complex physical transformations?
The study, conducted by Cassidy Ashworth, Pietro Liò, and Francesco Caso, focuses on a cornerstone of theoretical physics: the renormalisation group (RG) transformation. From a probabilistic perspective, RG can be understood as a process that transforms distributions of physical properties. To explore this, the researchers used the Central Limit Theorem (CLT) as a simplified, yet powerful, test case. The CLT describes how the sum of many independent random variables, regardless of their original distribution, tends towards a Gaussian (bell-curve) distribution.
The core finding of this research highlights a delicate balance: a competition between imposing symmetry constraints on neural network parameters and allowing the network sufficient expressivity (its capacity to learn complex functions). The authors discovered that models that were either too complex or too rigidly constrained by symmetry often generalized poorly to new, unseen data. This suggests that while encoding physical symmetries can be beneficial, it must be done carefully and in alignment with the specific demands of the learning task.
Exploring Multilayer Perceptrons (MLPs)
The study first examined simple Multilayer Perceptrons (MLPs), which are foundational neural networks. They varied weight symmetries and activation functions (the mathematical functions that introduce non-linearity into the network) across different architectures. For linear networks, where the relationships are straightforward, symmetry constraints had little impact, and models generalized well. However, when non-linear activation functions like quadratic, ReLU, and Leaky ReLU were introduced, the picture became more complex.
For instance, networks with a quadratic non-linearity and symmetric weights showed an analytical inconsistency, indicating that such a setup couldn’t perfectly learn the CLT transformation, leading to poor generalization. Similarly, ReLU activations, known for their strong non-linearity, performed even worse when combined with strict symmetry constraints. This suggests that in these cases, the network needed to “break” some symmetry to effectively learn the transformation.
Interestingly, Leaky ReLU networks with unconstrained (asymmetric) weights exhibited a “phase transition-like phenomenon.” Weak non-linearities initially led to poor generalization, but performance significantly improved as the degree of non-linearity increased. Conversely, when Leaky ReLU networks had symmetric weights, they displayed “frustrated learning dynamics,” where competing symmetries hindered optimal learning. The research also looked at spline activations, which are learnable non-linearities. When the network’s weights were fixed, the spline tended to overfit. But when both weights and spline parameters were trainable, the model effectively simplified itself to a linear architecture, performing much better. This implies that networks, when given the freedom, prefer to minimize unnecessary non-linearity if the task doesn’t demand it.
Investigating Graph Neural Networks (GNNs)
The researchers also extended their analysis to Graph Neural Networks (GNNs), which are designed to handle structured data like graphs. They applied GNNs to simple two-node directed graphs, mirroring the two-to-one dimensionality reduction of the CLT. A significant part of this work involved extending an existing framework that tracks how statistical properties (cumulants) propagate through MLP layers to these more complex GNN architectures. While this framework successfully tracked low-order cumulants (like mean and variance) to a good extent, it struggled with higher-order cumulants, primarily because it approximated nodes as independent, neglecting crucial correlations that emerge during message passing in GNNs.
Despite their specialized inductive biases for structured data, GNNs performed comparably to MLPs on this very simple graph task. The study concluded that for such minimal graph structures, the GNN’s built-in permutation equivariance (its ability to produce consistent outputs regardless of how nodes are ordered) wasn’t fully exploited, and in some cases, its architectural biases might have even hindered learning compared to a simpler MLP.
Also Read:
- Unveiling Sparse Network Training: A Graphon Perspective on Neural Network Pruning
- Unveiling Hidden Patterns in Directed Percolation with Deep Learning
A New Lens for Understanding Learning
The analytical framework developed in this paper, which tracks cumulant propagation through network layers, offers a clear way to interpret how neural networks process information and learn physically meaningful transformations. It provides insights into the internal workings of these models, moving beyond just observing their input-output behavior.
In conclusion, this research underscores the importance of a critical balance between symmetry constraints and network expressivity in designing effective neural networks for physics applications. Symmetry should be leveraged carefully, aligning with the task’s representational needs, rather than being overly restrictive. Conversely, excessively flexible architectures can also lead to reduced performance due to overfitting or a mismatch with the task’s inherent structure. These findings offer valuable guidance for future neural network design, suggesting that the architectural bias of a model must be thoughtfully matched to the specific task at hand. For more detailed information, you can read the full paper here.
