TLDR: A new study reveals that Large Language Models (LLMs) can learn to predict the dynamics of physical systems through in-context learning. By analyzing the LLM’s internal representations using Sparse Autoencoders, researchers found that the models spontaneously encode key physical concepts like energy, which are crucial for accurate predictions. The study demonstrates that LLMs don’t just match patterns but can implicitly understand and leverage fundamental physics principles.
Large Language Models (LLMs) have shown remarkable abilities to learn and perform tasks simply by being given examples within a text prompt, a phenomenon known as in-context learning (ICL). This capability has expanded to various domains, from basic math to time-series prediction. However, understanding the precise internal mechanisms that allow LLMs to succeed across such diverse tasks has remained a significant challenge.
A recent research paper, titled “Uncovering Emergent Physics Representations Learned In-Context by Large Language Models,” delves into this mystery by using physics-based tasks as a unique testing ground. Unlike synthetic data, physical systems offer real-world, structured data based on fundamental principles, making them ideal for probing how LLMs develop reasoning behaviors in a realistic yet controlled environment.
The study, conducted by Yeongwoo Song, Jaeyong Bae, Dong-Kyum Kim, and Hawoong Jeong, investigates whether LLMs can learn physics in context. They specifically focused on a dynamics forecasting task, where an LLM was prompted to predict the future behavior of physical systems based on their past trajectories.
How the Study Was Conducted
The researchers used Qwen3, a large language model, and tasked it with forecasting the dynamics of two types of physical systems: coupled mass-spring oscillators (a relatively simple system) and coupled pendulums (a more chaotic and challenging system). The LLM was given historical trajectory data and asked to predict subsequent time steps without any explicit instructions or fine-tuning.
To understand what was happening inside the LLM, the team employed a technique called Sparse Autoencoders (SAEs). SAEs are tools that help to disentangle the complex internal representations (called residual stream activations) of neural networks into more interpretable, ‘monosemantic’ features. By analyzing these features, the researchers aimed to see if recognizable physical quantities, such as energy, emerged within the LLM’s internal processing.
They performed two main types of analysis: correlation analysis and intervention studies. In the correlation analysis, they checked if the sparse activations from the SAEs correlated with key physical variables like total energy, kinetic energy, and potential energy. For the intervention study, they selectively removed (ablated) the sparse activations that showed high correlations with energy and observed the impact on the LLM’s prediction accuracy.
Key Findings
The study yielded several significant insights into how LLMs learn physics in context:
-
Improved Accuracy with Longer Context: The LLM’s ability to forecast physical dynamics improved significantly as it was provided with longer historical input (context length). This suggests that more data helps the model build a better internal understanding of the system.
-
Emergence of Energy-Correlated Features: The analysis using Sparse Autoencoders revealed that certain internal features within the LLM’s residual stream activations strongly correlated with physical energy quantities (total, kinetic, and potential energy). This indicates that the model wasn’t just memorizing patterns but was implicitly encoding meaningful physical concepts.
-
Context-Dependent Learning: The emergence of these energy-correlated representations was found to be context-dependent. For the simpler mass-spring system, these correlations were robust even with shorter context lengths. However, for the more chaotic coupled pendulum system, stronger correlations with kinetic energy only emerged with longer contexts, suggesting that more complex systems require richer historical input for the model to capture their underlying physics.
-
Functional Importance of Energy Representations: The intervention experiments provided compelling evidence that these energy-related representations are functionally important for accurate predictions. When the highly energy-correlated sparse activations were removed, the LLM’s prediction accuracy significantly degraded, especially for the pendulum system at longer context lengths. This confirms that the LLM leverages this implicit understanding of energy to make its forecasts.
Also Read:
- Unveiling Survival Instincts in Large Language Model Agents
- Unpacking AI’s Explanations: Why Predicting Word Features Isn’t Always Understanding
Implications for AI and Physics
This research provides a novel case study that broadens our understanding of how LLMs learn in context. It suggests that these models can implicitly encode fundamental physical dynamics and even recover abstract concepts like energy, which are foundational inductive biases in physics. This mirrors how humans distill complex dynamics into conserved quantities.
The findings open up exciting avenues for future research, including leveraging physics-informed tasks to further understand the internal mechanisms of large-scale models and advancing the design of LLM-based agents capable of reasoning and acting in physically realistic environments. You can read the full research paper here: Uncovering Emergent Physics Representations Learned In-Context by Large Language Models.
