TLDR: A new research paper investigates how Large Language Models (LLMs) handle numbers internally, revealing that despite output errors, LLMs learn systematic, accurate, and universal sinusoidal representations of numbers across their hidden states and contexts. The study demonstrates that specialized probes can accurately extract numerical information and trace errors to specific layers, offering a fundamental understanding of LLM numerical manipulation and potential for architectural refinements.
Large Language Models (LLMs) have become incredibly powerful, but they often struggle with numbers, frequently producing incorrect outputs when dealing with numerical information. This can seem contradictory, especially since recent research suggests that LLMs actually develop very accurate internal representations for numbers. A new study aims to unravel this mystery, exploring how these models handle numbers internally and quantifying the accuracy of these mechanisms. The findings offer a deeper understanding of how LLMs process numerical data and open doors for more precise ways to identify and fix errors.
The research reveals that despite the errors we sometimes see in their outputs, different LLMs learn remarkably similar and highly accurate ways to represent numbers. These representations are systematic and universal, appearing consistently across the models’ internal layers and various types of input contexts. This consistency allows for the creation of “universal probes” – tools that can accurately extract numerical information from any LLM and even pinpoint where errors originate within the model’s processing layers. This work provides a foundational understanding of how pre-trained LLMs manipulate numbers and suggests improved techniques for refining their architectures.
The Universal Language of Numbers in LLMs
One of the core discoveries is that LLMs, regardless of their size or family (like OLMo, Llama, and Phi models), converge on the same type of numerical representation. This was demonstrated using techniques like Representational Similarity Analysis (RSA) and Fourier decompositions, which showed that number embeddings consistently have higher similarity scores and share dominant “frequential” characteristics compared to random words. Essentially, numbers are encoded in a “sinusoidal” or wave-like pattern within the models.
This sinusoidal representation isn’t just present at the input stage; it’s maintained and utilized throughout the model’s internal workings. The study found that specialized “sinusoidal probes” were far more effective at decoding numerical values from the models’ hidden layers than other types of probes. This high accuracy confirms the pervasive sinusoidal nature of number representations across layers and different contexts, including complex natural language scenarios like mathematical word problems, date extraction, medical codes, and recipe ingredients. Interestingly, probes trained on natural language contexts proved to be more robust and generalized better to various situations, highlighting the importance of real-world data for understanding model mechanics.
Numbers Across Layers and Error Tracking
The research further explored how these numerical representations evolve as information passes through different layers of an LLM. It was found that the sinusoidal structure remains largely consistent across intermediate layers, meaning a probe trained on one layer can accurately decode numerical information from many other layers. This “cross-layer generalization” suggests that LLMs operate with a consistent internal representation of numbers throughout most of their computation. While input and output layers showed some differences in how “sparse” or distributed these features were, the overall consistency is primarily maintained by what are called “residual streams” – pathways that carry information directly across layers.
Understanding these internal mechanisms also provides a powerful way to track errors. The study showed that the internal representations of LLMs can predict with high accuracy whether the model will produce a correct or incorrect answer for arithmetic operations. For addition and subtraction, the internal representations often held the correct result even when the final output was wrong. More strikingly, the research identified specific layers that were responsible for “breaking” the correct result from previous layers. In some cases, simply removing these problematic layers led to significant error reductions (e.g., 27-64% in division tasks). This ability to pinpoint and potentially mitigate error sources is a major step towards building more reliable LLMs for numerical reasoning.
Also Read:
- Decoding Relational Knowledge in Large Language Models: A Property-Based Approach
- Decoding How Large Language Models Filter Information
Handling Multi-Token Numbers
The paper also investigated how LLMs represent numbers that span multiple tokens (e.g., “1,234” might be tokenized as “1”, “,”, “234”). It was discovered that multi-token numbers are systematically and accurately combined into a single representation, especially for immediately preceding tokens. For example, the last numeric token could accurately represent the value of the preceding token with 99% accuracy. However, this accuracy decreased for very long numbers (e.g., those requiring five or more tokens), suggesting limitations in how models superpose very large numerical values. This superposition mechanism isn’t unique to numbers but appears to be more prevalent and accurate for them.
In conclusion, this groundbreaking research provides a fundamental understanding of how LLMs internally represent and manipulate numbers. It establishes that diverse LLMs learn and utilize consistent, interchangeable sinusoidal representations across their internal layers. By leveraging this knowledge, more accurate probing techniques can be developed to track the causes of errors within a model’s internal computations, paving the way for more robust and faithful reasoning models. For more details, you can read the full paper here.
