TLDR: A new research paper, “Deprecating Benchmarks: Criteria and Framework,” proposes a systematic approach to retire or update outdated and flawed AI benchmarks. It identifies issues like benchmark saturation, data contamination, and task obsolescence that lead to misleading evaluations. The framework outlines three phases—assessment, reporting, and notification—to ensure transparent and effective deprecation. The goal is to improve the quality and reliability of AI evaluations, particularly for frontier models, benefiting developers, users, and governance bodies by preventing inflated performance claims and addressing safety concerns.
As artificial intelligence (AI) models rapidly advance, benchmarks play a crucial role in comparing different models and measuring their progress across various tasks. However, a significant challenge has emerged: a lack of clear guidance on when and how these benchmarks should be retired or updated once they no longer serve their intended purpose effectively. This oversight risks overstating model capabilities or, worse, obscuring potential safety issues, a practice sometimes referred to as ‘safety-washing’.
A new research paper, “Deprecating Benchmarks: Criteria and Framework”, addresses this critical gap. Based on a comprehensive review of current benchmarking practices, the authors propose a set of criteria to determine when benchmarks should be fully or partially deprecated, along with a practical framework for the deprecation process. This work aims to foster more rigorous and high-quality evaluations, especially for cutting-edge AI models, benefiting benchmark developers, users, AI governance bodies, and policymakers alike.
Why Deprecate Benchmarks? The Core Issues
Many benchmarks currently in use are outdated, flawed, or misaligned with their original goals. Some persist simply due to historical ties to influential models, becoming de facto standards despite their limitations. Commercial incentives further complicate this, as AI labs may be reluctant to deprecate benchmarks that favorably showcase their models, potentially inflating performance claims without genuine progress.
The paper identifies several key reasons why benchmarks need active deprecation:
- Saturation: Benchmarks reach a point where models perform so well that further improvements offer little new information about true capabilities. Examples include MMLU and GSM8K.
- Contamination: Models memorize benchmark data due to leakage, meaning their performance doesn’t reflect true generalization.
- Statistical Bias: Poorly balanced datasets can skew results, allowing models to exploit imbalances rather than demonstrating intended capabilities.
- High Annotation Error Rate: Errors introduced during data labeling compromise benchmark quality and lead to inaccurate evaluations. For instance, a significant portion of MMLU’s Virology subset was found to contain errors.
- Task Obsolescence: The underlying task measured by a benchmark may no longer be relevant or has effectively been solved, leading to misleading signals of progress.
- Invalidated Assumptions: Benchmarks often rely on simplifying assumptions that become inappropriate as the field evolves, failing to reflect real-world complexity. The “Needle-in-a-Haystack” test for LLMs is cited as an example, as it doesn’t fully represent complex retrieval-augmented generation (RAG) scenarios.
- Semantic Drift: Over time, the meaning or interpretation of a task or its labels can change, making benchmarks outdated or unrepresentative.
A Framework for Deprecation
The proposed deprecation framework consists of three distinct phases:
1. Assessment: This initial phase determines if deprecation is necessary by evaluating potential impacts and identifying which benchmark components remain valid. It involves regularly tracking performance curves, reviewing emerging critiques in literature, and soliciting feedback from user communities. The framework also suggests establishing a formal appeals process, allowing developers to contest deprecation decisions or propose partial deprecation (updating) instead of full retirement.
2. Reporting: If deprecation is deemed necessary, a comprehensive deprecation report is created. This report must clearly state the rationale and risks behind the decision, supported by evidence from the assessment phase. It provides explicit instructions for future benchmark usage, distinguishing between full and partial deprecation. For updates, it clarifies which components remain valid and how past results should be interpreted. The report also outlines implementation timelines, mitigation strategies, and suggests alternative benchmarks or updated versions.
3. Notification: The final step ensures that users are informed to prevent continued use of deprecated benchmarks and minimize disruption. Deprecation notices should ideally appear in the same channels as the original benchmark publication. Clear visual indicators or metadata should distinguish deprecated benchmarks from active ones in catalogs, similar to academic retraction notices. Direct notification is recommended for key users whose evaluations might be invalidated, especially in system-critical settings.
Adapting the Framework in Practice
The paper illustrates how this framework could be adapted, using the European Union’s AI Office (AIO) as an example. The AIO could compile and periodically review benchmarks for safety-critical tasks, determining which require deprecation. This would involve inspecting model cards, weighing research critiques, directly testing models, and convening expert consultations. The AIO would then create deprecation reports and notify member states and affected AI programs, requiring models deployed commercially in the EU to update their model cards and technical reports accordingly.
Also Read:
- Unmasking AI Vulnerabilities: A New Framework for Trustworthy Robustness Evaluation
- Establishing a Scientific Foundation for Measuring Artificial Intelligence
Key Recommendations
The authors provide several recommendations for benchmark developers, policymakers, and governance actors:
- Establish Version Control: Benchmarks need clear version control, ideally using Digital Object Identifiers (DOIs), to track changes and prevent invalid comparisons.
- Have a Deprecation Plan: Developers should create a deprecation plan when designing a benchmark, including archiving related artifacts.
- Establish a Point of Contact: A designated contact person should handle queries related to deprecated benchmarks.
- Create Context-Specific Deprecation Lists: Since benchmarks are not universal, deprecation lists should consider the specific context of evaluation.
- Construct Lists Transparently: Deprecation lists must be built on verifiable claims and thorough analysis.
- Clearly Communicate Reasons: The rationale for including a benchmark on a deprecation list must be clearly communicated to maintain trust.
- Discourage Use of Deprecated Benchmarks: Compliance requirements should discourage the use of problematic benchmarks, accelerating their retirement.
- Establish an Appeal Mechanism: An appeal process promotes trust and ensures responsiveness to feedback from benchmark developers.
By implementing these criteria and the proposed framework, the AI community can move towards a more robust and reliable evaluation ecosystem, ensuring that benchmarks accurately reflect the true capabilities and safety properties of rapidly evolving AI models.
