TLDR: DISCO is a novel method for efficiently evaluating machine learning models by selecting a small, informative subset of evaluation data. Instead of focusing on sample diversity, DISCO identifies samples that maximize disagreement among models. It then uses ‘model signatures’ (concatenated outputs on this subset) with simple predictors to estimate full benchmark performance. This approach significantly reduces evaluation costs (over 99%) with minimal error, outperforming prior methods across language and vision domains, making model evaluation faster and more accessible.
Evaluating modern machine learning models has become an incredibly expensive and time-consuming task. Benchmarks for large language models (LLMs) and other AI systems often demand thousands of GPU hours, which can hinder innovation, reduce inclusivity in research, and increase environmental impact. Traditionally, researchers select a small ‘anchor’ subset of data and then train a system to predict a model’s full performance based on its accuracy on this subset. However, this anchor selection often relies on complex clustering methods that can be difficult to manage and sensitive to specific design choices.
A new research paper introduces a novel approach called DISCO: Diversifying Sample Condensation for Efficient Model Evaluation. This method challenges the conventional wisdom that diversity among data samples is key. Instead, DISCO argues that what truly matters is selecting samples that maximize diversity in how models respond to them. In simpler terms, it focuses on the samples where different models are most likely to disagree.
How DISCO Works
DISCO simplifies the evaluation process into two main steps:
1. Dataset Selection: Instead of complex clustering, DISCO identifies the most informative samples by looking for those that generate the greatest disagreement among a set of reference models. This is achieved using a metric called Predictive Diversity Score (PDS) or Jensen-Shannon Divergence (JSD). These scores help pinpoint samples where models produce varied or conflicting outputs, which the researchers prove to be the most informative for predicting overall model performance. This greedy, sample-wise selection is conceptually much simpler than global clustering.
2. Performance Prediction: Once a small, informative subset of samples is selected, DISCO uses the target model’s raw outputs on these specific samples to estimate its full benchmark performance. These concatenated outputs are called ‘model signatures’. Instead of trying to estimate hidden model parameters, DISCO feeds these high-dimensional model signatures directly into simple prediction models, such as Random Forests or k-Nearest Neighbors (kNN). This direct approach is shown to be more effective and less complex than prior methods.
Key Advantages and Results
DISCO offers significant improvements over existing efficient evaluation methods. It achieves state-of-the-art results in performance prediction across various benchmarks, including MMLU, Hellaswag, Winogrande, and ARC. For instance, on the MMLU benchmark, DISCO reduced evaluation costs by an impressive 99.3% with only a 1.07 percentage point error in accuracy prediction. This means that models can be evaluated using a tiny fraction of the original test data, drastically cutting down on computational resources and time.
The method’s effectiveness stems from its focus on model disagreement as a proxy for informativeness, which is a simpler and more powerful signal than traditional ‘representativeness’ criteria. Furthermore, using model signatures for direct performance prediction avoids the complexities of psychometric modeling found in other approaches.
DISCO has been validated in both language and vision domains, demonstrating its versatility. The researchers also showed that DISCO is robust to different ways of splitting models for training and testing, including a ‘chronological split’ that better reflects real-world scenarios where newer models are evaluated against older ones.
Also Read:
- Enhancing Trust in Large Language Models with Domain-Shift-Aware Uncertainty
- Enhancing LLM Judge Reliability Through Adaptive Prompting and Collective Confidence
Impact and Future
By making model evaluation vastly more efficient, DISCO enables practical applications such as frequent performance tracking during model training, cost-effective checks of deployed models, and more accessible research for those with limited computational resources. This innovation could significantly accelerate the cycle of AI innovation and reduce its environmental footprint.
For more technical details, you can read the full research paper here: DISCO: Diversifying Sample Condensation for Efficient Model Evaluation.
