TLDR: A new Bayesian framework, DON’TPASS@k, is proposed to improve Large Language Model (LLM) evaluation by replacing traditional Pass@k and average accuracy metrics. This framework uses posterior estimates of success probabilities and credible intervals, modeling outcomes as categorical with a Dirichlet prior. It offers faster convergence, greater rank stability, and explicit uncertainty quantification, enabling reliable LLM comparisons with fewer computational trials and clarifying statistically meaningful performance differences.
Evaluating the performance of Large Language Models (LLMs) is a critical step in their development and deployment. However, a widely used metric called Pass@k often falls short, leading to unstable and sometimes misleading rankings, especially when the number of trials or computational resources are limited. This can make it difficult to accurately compare different LLMs and understand their true capabilities.
A new research paper titled “DON’TPASS@k: A BAYESIAN FRAMEWORK FOR LARGE LANGUAGE MODEL EVALUATION” introduces a principled Bayesian evaluation framework designed to overcome these limitations. Authored by Mohsen Hariri, Amirhossein Samandar, Michael Hinczewski, and Vipin Chaudhary from Case Western Reserve University, this framework proposes a more robust and transparent way to assess LLM performance.
Moving Beyond Pass@k
The core issue with Pass@k and similar metrics like average accuracy over N trials (avg@N) is their instability. Imagine trying to rank several LLMs based on a small number of attempts; the results can fluctuate wildly, making it hard to trust the leaderboard. The new Bayesian framework replaces these with posterior estimates of an LLM’s underlying success probability and provides credible intervals. These intervals offer a clear measure of uncertainty, helping evaluators understand when observed differences between models are statistically meaningful versus just random noise.
Key Innovations of the Bayesian Framework
One of the significant advancements is how evaluation outcomes are modeled. Instead of simply binary (correct/incorrect), the framework treats outcomes as categorical. This means an LLM’s response can be classified into multiple levels, such as correct, partially correct, formatting error, or refusal. By using a Dirichlet prior, the framework provides closed-form expressions for the posterior mean and uncertainty for any weighted rubric. This allows for a more nuanced and flexible evaluation process.
The framework addresses four persistent challenges in LLM evaluation:
- Convergence: It helps methods converge to the true underlying ranking with fewer trials, meaning less computational effort is needed to get reliable results.
- Credible Intervals: It provides a simple rule: if the credible intervals of two models overlap, don’t declare a winner. This reduces leaderboard churn and prevents over-interpreting small performance gaps.
- Categorical Evaluation: It unifies binary and non-binary evaluations, making it natural to incorporate graded rubrics for assessing step-by-step reasoning or partial credit.
- Prior Information: The framework can incorporate existing knowledge or evidence from previous evaluations, potentially accelerating convergence even further.
Empirical Validation and Real-World Impact
The researchers validated their approach through controlled simulations with known ground-truth success rates and on real-world math reasoning benchmarks, including AIME’24/’25, HMMT’25, and BrUMO’25. In these experiments, the Bayesian procedure consistently achieved faster convergence and greater rank stability compared to Pass@k and its recent variants. This means reliable comparisons can be made with significantly fewer samples, saving valuable compute resources.
For instance, on datasets like HMMT’25 and BrUMO’25, the Bayesian method achieved stable rankings with approximately 44.2 and 27.1 trials, respectively, while Pass@k methods required around 69.5 and 48.5 trials. This demonstrates a substantial improvement in efficiency.
The framework also clarifies when differences between models are statistically significant. By incorporating 95% confidence intervals, the rankings reveal ties between models whose performance differences are too small to be confidently distinguished with the given number of trials. This prevents premature conclusions about model superiority.
Also Read:
- BayesianRouter: A Smart Approach to Aligning Language Models with Human Preferences
- Automating Prompt Engineering with Bayesian Optimization for Enhanced LLM Performance
Future Directions and Reproducibility
While the current work focuses on a simple version with a uniform prior, the theory allows for more complex, informative priors. Future research could explore using priors from past runs or domain-specific knowledge to further accelerate convergence. The authors emphasize the importance of clear documentation and reporting when using user-defined priors to prevent potential misuse like cherry-picking to exaggerate performance.
This new Bayesian framework offers a powerful, compute-efficient, and transparent protocol for LLM evaluation, unifying binary and non-binary assessments while explicitly accounting for uncertainty. It promises to make LLM leaderboards more stable and trustworthy, guiding better decisions in model adoption and resource allocation. For more details, you can read the full research paper here.
