TLDR: A new research paper introduces a decomposition of cross-entropy loss into three components: Error-Entropy, Self-Alignment, and Confidence. The study reveals that only Error-Entropy consistently follows a robust power-law scaling as large language models grow in size. This ‘Error-Entropy Scaling Law’ explains why the traditional cross-entropy scaling law appears accurate for smaller models but breaks down at very large scales, as Error-Entropy’s proportion of the total loss diminishes. This finding offers a more precise understanding of model behavior and could guide future LLM development.
For a long time, the cross-entropy scaling law has been a guiding principle in the development of large language models (LLMs). This law suggested that as models grew larger, their cross-entropy loss would decrease in a predictable, power-law fashion. This understanding has been crucial for tasks like balancing model parameters with data size, predicting performance, and fine-tuning training settings. However, recent observations have cast doubt on this law, especially at very large scales, where the loss decreases more slowly than anticipated, posing significant challenges for advancing LLMs.
A new research paper, titled WHAT SCALES IN CROSS-ENTROPY SCALING LAW?, proposes a fresh perspective on this issue. The authors, Junxi Yan, Zixi Wei, Jingtao Zhan, Qingyao Ai, and Yiqun Liu from Tsinghua University, hypothesize that the problem isn’t with the scaling law itself, but with the cross-entropy loss. They suggest that cross-entropy itself doesn’t truly scale; instead, only a hidden component within it does.
Deconstructing Cross-Entropy Loss
To investigate this, the researchers introduce a novel way to break down cross-entropy into three distinct parts: Error-Entropy, Self-Alignment, and Confidence. This decomposition offers a clearer view of how models learn and optimize during training.
-
Error-Entropy: This term measures the ‘entropy’ of errors made by the model. Essentially, it quantifies how uncertain the model is about the correct answer. Minimizing Error-Entropy means the model is getting better at ranking the correct token higher among its predictions, making its predictions more accurate.
-
Self-Alignment: This component looks at how well the model’s predicted probability scores align with its internal error distribution. It suggests that models assign probabilities based on their own likelihood of making errors, rather than strictly representing a universal language distribution.
-
Confidence: This term reflects the overall magnitude or ‘confidence’ of the model’s probability scores for correct tokens. A higher confidence term indicates that the model is more certain about its predictions, assigning very low probabilities to tokens it believes are incorrect.
The study found that during training, all three components are optimized. Error-Entropy and Self-Alignment decrease, while the Confidence term increases. Interestingly, Error-Entropy, being the largest initially, is optimized first, followed by the other two, which are smaller in magnitude.
The Error-Entropy Scaling Law
Through extensive experiments involving multiple datasets and 32 models of varying sizes, the researchers made a significant discovery: only Error-Entropy consistently follows a robust power-law scaling. The other two terms, Self-Alignment and Confidence, largely remain invariant or show inconsistent patterns as model size increases.
This finding is crucial because it suggests that Error-Entropy is the true driver behind the observed scaling behavior in LLMs. In fact, the power-law fit for Error-Entropy was often even better than that for the traditional cross-entropy loss, indicating it’s a more accurate description of how models improve with scale.
Explaining the Scaling Law’s Breakdown
The new Error-Entropy scaling law also provides a clear explanation for a long-standing puzzle: why the cross-entropy scaling law works well for smaller models but falters for very large ones. The answer lies in the proportion of Error-Entropy within the total cross-entropy loss.
For smaller models, Error-Entropy constitutes a dominant share (around 80-90%) of the cross-entropy loss. Because Error-Entropy scales predictably, the overall cross-entropy loss appears to follow a clean power law. However, as models grow larger, the proportion of Error-Entropy gradually declines. The non-scaling components, Self-Alignment and Confidence, then take up a larger share, causing the total cross-entropy loss to deviate from the expected power-law trend. This explains why the scaling law appears to break down at very large scales.
Also Read:
- How AI Models Express Their Confidence: A Look at Uncertainty in Argumentative Language Models
- Unlocking Latent Reasoning in LLMs with Temperature Scaling
Future Implications
The decomposition of cross-entropy and the discovery of the Error-Entropy scaling law offer a more fundamental understanding of how language models truly scale. This new framework could have wide-ranging applications, from guiding the design and training of future large language models to probing their internal mechanisms and even contributing to fundamental theories of artificial intelligence. For instance, it might lead to new training objectives that focus solely on optimizing the Error-Entropy term.
