TLDR: A new research paper introduces the Relative-Based Scaling Law, which uses a metric called Relative-Based Probability (RBP) to evaluate how language models improve with size. Unlike traditional cross-entropy, RBP focuses on the rank of the correct token among predictions, which is crucial for practical applications like greedy decoding. The law demonstrates that as models scale, the likelihood of the correct token being highly ranked increases predictably. This new law helps explain emergent abilities in LLMs and suggests a deeper connection between different scaling phenomena, paving the way for a unified theory of AI scaling.
In the rapidly evolving world of artificial intelligence, understanding how large language models (LLMs) improve as they grow in size is crucial. This understanding comes from what are known as scaling laws, which predict model performance across different scales. Traditionally, these scaling laws have relied almost exclusively on a metric called cross-entropy to evaluate performance. However, a recent research paper titled “RELATIVE-BASED SCALING LAW FOR NEURAL LANGUAGE MODELS” by Baoqing Yue, Jinyuan Zhou, Zixi Wei, Jingtao Zhan, Qingyao Ai, and Yiqun Liu from Tsinghua University, introduces a new perspective that offers a more complete picture.
The Limitations of Cross-Entropy
Cross-entropy is a valuable metric, but it primarily measures the absolute probability assigned to the correct token. Think of it this way: if a model predicts a word with a high probability, cross-entropy reflects that. However, it doesn’t tell us where that correct word ranks among all the other possible predictions. For instance, a model might assign a 0.28 probability to the correct word “Paris,” but if two incorrect words like “London” (0.30) and “Beijing” (0.29) have even higher probabilities, then “Paris” is ranked third. In practical applications like greedy decoding (where the model simply picks the top-ranked word) or top-k sampling, this relative ordering is incredibly important. If the correct word isn’t ranked highly, the model’s real-world performance can suffer, even if its cross-entropy score looks good.
Introducing Relative-Based Probability (RBP)
To address this gap, the researchers propose a new metric called Relative-Based Probability (RBPk). This metric quantifies the probability that the correct token is ranked among the model’s top-k predictions. For example, RBP1 would tell you the probability that the correct token is the very top prediction, which is directly relevant to greedy decoding. RBP10 would tell you the probability that the correct token is within the top 10 predictions, relevant for top-k sampling. This offers a complementary view to cross-entropy, focusing on the practical utility of a model’s predictions rather than just their absolute likelihoods.
The Relative-Based Scaling Law
Building on RBP, the paper establishes the Relative-Based Scaling Law. This law characterizes how RBP improves as model size increases. Specifically, for small values of ‘k’ (where ‘k’ is much smaller than the total vocabulary size), the negative logarithm of RBPk scales inversely with model size raised to a positive constant (S^-alpha). In simpler terms, as language models get larger, the probability of the correct token being ranked among the top predictions increases predictably, following a power-law trend. This means that scaling up models fundamentally reshapes the ranking of tokens, leading to more correct tokens appearing at the top of the prediction list.
Robust Experimental Validation
The researchers conducted extensive experiments across four diverse datasets (Wikipedia, HumanEval, HotpotQA, and Open Australian Legal Corpus) and four model families (Pythia, GPT-2, OPT, and Qwen), spanning five orders of magnitude in model size. The results consistently showed that for small and moderate ‘k’ values (roughly 1 to 100), the Relative-Based Scaling Law holds with high accuracy, achieving R-squared values comparable to those of cross-entropy-based scaling laws (around 0.99). However, the law’s predictive power diminishes when ‘k’ approaches the entire vocabulary size, as random noise begins to dominate the metric.
Applications and Deeper Insights
The Relative-Based Scaling Law offers two significant applications:
1. Explaining Emergence Phenomena: The concept of “emergence” in LLMs refers to the sudden, non-linear jumps in performance on certain tasks once a model reaches a certain size. Previous explanations using cross-entropy struggled to account for emergence in scenarios like greedy or top-k sampling. The Relative-Based Scaling Law provides a natural explanation: emergence is not a breakdown of scaling but a predictable macroscopic effect of smooth power-law scaling at the token level, amplified by sequence length. The probability curve for task success (pN,k = (RBPk)N) inherently exhibits a sigmoidal shape, creating the observed sharp transitions.
2. Connecting Scaling Laws: Surprisingly, the Relative-Based Scaling Law (especially for k=1) exhibits a very similar mathematical form and decay exponent to the cross-entropy-based scaling law. This intriguing coincidence suggests a deeper, unified theory of intelligence that could explain both phenomena simultaneously. The paper proposes a conjecture based on a lognormal distribution of token ranks, which, when simulated, successfully reproduces the similar scaling behaviors of both metrics. This offers a new probabilistic framework to connect these previously separate aspects of model performance.
Also Read:
- Improving LLM Judge Reliability with Contrastive Decoding
- Neural Diversity: A Novel Approach to Curb Language Model Hallucinations
A More Complete Understanding
In conclusion, the Relative-Based Scaling Law complements the traditional cross-entropy perspective, providing a crucial relative-ordering view of how neural language models scale. It offers valuable insights for both practical development, by better reflecting real-world decoding strategies, and theoretical exploration, by deepening our understanding of phenomena like emergence and hinting at a unified theory of scaling. For more details, you can read the full research paper here.
