TLDR: Scales++ is a new method for efficiently evaluating large language models (LLMs) by selecting small, representative data subsets. Unlike traditional “model-centric” approaches that rely on past model performance, Scales++ uses an “item-centric” paradigm, selecting items based on their intrinsic cognitive demands (e.g., logical reasoning, knowledge areas). This reduces upfront evaluation costs by over 18x, solves the “cold-start” problem for new models, and provides more interpretable results. Scales++ achieved a 2.9% mean absolute error on the Open LLM Leaderboard using only 0.5% of the data. A “Lite” version further reduces annotation costs using a Graph Neural Network, enabling rapid annotation of large datasets.
Evaluating large language models (LLMs) is a crucial but increasingly expensive task. As these models grow in size and complexity, running full evaluations on comprehensive benchmarks demands significant computational resources and time. This high cost has led researchers to seek methods for creating smaller, yet representative, data subsets – often called “tiny benchmarks” – that can efficiently assess LLM performance while still accurately predicting how they would perform on the full dataset.
Traditionally, many approaches to creating these tiny benchmarks have been “model-centric.” This means they select benchmark items based on the past performance and failure patterns of existing models. While these methods have shown some success, they come with notable drawbacks. They incur large upfront costs because they require evaluating many models on the full dataset first. They also struggle with “cold-start” scenarios, where no historical data is available for new or private model families. Furthermore, they operate under the fragile assumption that future models will exhibit similar failure patterns to their predecessors, which isn’t always reliable.
A New Item-Centric Approach: Scales++
A new research paper, “Scales++: Compute Efficient Evaluation Subset Selection with Cognitive Scales Embeddings,” challenges this model-centric paradigm. The authors propose an “item-centric” approach, arguing that benchmark subset selection should be based on the inherent properties of the task items themselves, rather than relying on how models have performed on them in the past. This novel method, called Scales++, focuses on the cognitive demands of the benchmark samples.
Scales++ works by annotating each benchmark item along 16 cognitively grounded dimensions. These dimensions cover various cognitive skills and knowledge areas, such as logical reasoning, attention and scan, and knowledge of specific scientific or social domains. These annotations create a unique “cognitive scales embedding” for each item. Instead of using historical model performance, Scales++ selects a small, diverse subset of items based on these cognitive demand embeddings. It then predicts full-benchmark performance using a combination of cluster-weighted estimates and per-dimension predictors.
Key Advantages and Efficiency
This item-centric approach offers several significant advantages. Firstly, it dramatically reduces upfront selection costs. The paper demonstrates that Scales++ cuts these costs by over 18 times compared to traditional methods. Secondly, it inherently solves the “cold-start” problem, as it doesn’t require any historical model performance data. This makes it particularly valuable for evaluating new or private model families where such data is unavailable. Thirdly, the cognitive demand embeddings provide a more interpretable way to understand what makes certain benchmark items challenging.
On the Open LLM Leaderboard, Scales++ achieved impressive results. Using just a 0.5% data subset, it predicted full benchmark scores with a mean absolute error of only 2.9%. This performance is competitive with, and in some cases surpasses, model-centric baselines, all while being significantly more efficient.
Also Read:
- Enhancing In-Context Learning for New Tasks with Graph-Based Pseudo-Labeling
- Improving LLM Reliability Through Semantic Confidence Rewards
Scales++ Lite: Further Cost Reduction
To make the annotation process even more scalable and cost-effective, the researchers also introduced Scales++ Lite. This variant uses a lightweight Graph Neural Network (GNN) predictor to estimate the 16-dimensional cognitive scales embeddings. This GNN is trained on a small auxiliary dataset with ground-truth GPT-4o annotations. Scales++ Lite can annotate the entire Open LLM Leaderboard, comprising 28,659 evaluation instances, in under 20 minutes. This further reduces computational requirements while maintaining competitive predictive accuracy, making efficient benchmarking accessible even for very large datasets.
In conclusion, Scales++ represents a fundamental shift in how we approach LLM evaluation. By focusing on the intrinsic cognitive demands of tasks, it provides a robust, efficient, and interpretable method for assessing LLM performance, overcoming the limitations of prior model-centric approaches and paving the way for more scalable and accessible LLM benchmarking.
