TLDR: A new research paper introduces “learnability” – a transformer model’s ability to extract structure from microscopic states – as a universal criterion for identifying phase transitions. Applied to the 2D Ising model, ordered phases showed enhanced learnability (reduced training loss, structured attention), while disordered phases resisted learning. Two unsupervised diagnostics, a sharp jump in training loss and a rise in attention entropy, accurately recovered the critical temperature, demonstrating a robust, data-driven method for charting phase diagrams and highlighting parallels between physical order and structure in language models.
Phase transitions are fundamental phenomena across science, marking dramatic qualitative changes in collective behavior, such as the shift from a paramagnet to a ferromagnet or the condensation of a fluid. While crucial for understanding many systems, identifying the exact boundaries of these transitions can be incredibly challenging, especially when traditional analytical solutions or simulations fall short. This is particularly true for complex systems like spin glasses or frustrated quantum magnets, where conventional methods often break down.
Historically, machine learning has offered alternative approaches to detect these transitions. Early methods included unsupervised techniques like Principal Component Analysis (PCA) and autoencoders. These models would analyze Monte Carlo generated configurations of physical systems, attempting to find patterns that signal a phase change. However, these methods have limitations; PCA is restricted to linear correlations, and autoencoders can produce latent spaces whose meaning isn’t always clear or directly aligned with physical order parameters.
A recent research paper, titled “Attention to Order: Transformers Discover Phase Transitions via Learnability,” by Şener Özönder, introduces a novel and universal criterion for identifying phase transitions: learnability. The core idea is simple yet profound: data from an ordered phase contains enough internal structure to be efficiently learned by a deep learning model, whereas disordered data resists such learning. This concept draws a direct parallel between the emergence of long-range order in condensed matter physics and the structured patterns found in human language that large language models learn.
The study specifically employs a transformer model, a type of neural network known for its attention mechanism, which excels at identifying context-dependent correlations. When applied to microscopic states of a physical system, this mechanism can uncover collective correlations among individual components. In an ordered phase, where long-range correlations are strong, the transformer rapidly reduces its training loss and develops distinct, structured patterns in its attention matrices. Conversely, in a high-temperature disordered phase, where correlations are weak, the model struggles to learn, resulting in persistently high training loss and disorganized attention patterns. Thus, the model’s very ability to learn from the data becomes a clear indicator of order-disorder transitions.
To demonstrate this, the researchers applied their method to the well-understood two-dimensional Ising model, a classic system for studying phase transitions. They generated spin configurations at various temperatures using Monte Carlo simulations. An encoder-only transformer was then trained from scratch on these configurations in a self-supervised manner, using a masked-language-modeling objective. This means the model was tasked with predicting masked spin values, forcing it to learn the underlying statistical regularities of the Ising model without any explicit temperature labels.
The study proposed two unsupervised diagnostics to pinpoint the critical temperature (Tc ≈ 2.27 for the 2D Ising model):
1. Sharp Jump in Training Loss
As the transformer was trained on configurations across different temperatures, a clear distinction emerged. For temperatures below the critical point, the model achieved significantly lower training loss, indicating successful learning of the ordered structure. Above the critical temperature, the loss remained uniformly high, reflecting the difficulty in learning from disordered states. The temperature at which this final training loss exhibited a sharp jump precisely matched the known critical temperature.
Also Read:
- Unlocking Long-Context Transformers: The Science Behind Attention Scaling
- Unlocking How Large Language Models Perceive Problem Difficulty
2. Rise in Attention Entropy
The attention matrices within the transformer, which quantify how strongly different spins attend to each other, also provided crucial insights. In ordered phases, attention concentrated on a sparse set of entries, reflecting strong, long-range correlations. In disordered phases, attention spread more uniformly, indicating a lack of strong correlations. By calculating the Shannon entropy of these attention heads, the researchers observed a pronounced increase across the phase boundary, providing an information-theoretic marker of the transition.
Both diagnostics independently recovered the critical temperature in excellent agreement with the exact analytical value. This approach is particularly powerful because it requires only raw microscopic configurations and does not rely on prior knowledge of an order parameter, making it suitable for complex systems where traditional tools are ineffective. The findings not only establish learnability as a robust, data-driven marker for phase transitions but also highlight deep connections between long-range order in physics and the emergence of structure in modern deep learning models. For more details, you can refer to the original research paper here.
