TLDR: A new research paper introduces Cover@τ, a metric designed to more accurately assess Large Language Model (LLM) reasoning capabilities by measuring problem-solving reliability, not just success rate. It argues that the traditional Pass@k metric can be misleading, especially at high sampling budgets, as it often conflates random guessing with genuine reasoning. Cover@τ offers a ‘breadth-depth’ view, revealing how consistently an LLM solves problems, and provides a more nuanced ranking of different reinforcement learning algorithms for LLMs.
In the rapidly evolving field of artificial intelligence, Large Language Models (LLMs) are increasingly being tasked with complex reasoning challenges, from coding to advanced mathematics. A key question for researchers is how to accurately measure an LLM’s ‘reasoning boundary’ – essentially, the range of problems it can reliably solve. Traditionally, a metric called Pass@k has been widely used, especially in conjunction with Reinforcement Learning with Verifiable Rewards (RLVR) techniques designed to enhance LLM performance.
However, recent research, including a new paper titled “Beyond Pass@k: Breadth-Depth Metrics for Reasoning Boundaries” by Marius Dragoi, Ioana Pintilie, Florin Gogianu, and Florin Brad from Bitdefender, Romania, argues that Pass@k can be misleading. The paper highlights a phenomenon where RLVR models might outperform base models at small ‘k’ values (meaning fewer attempts to solve a problem), but the base model often catches up or even surpasses them at very large ‘k’ values. This has been interpreted as the base model having a larger reasoning boundary.
The authors contend that for tasks with discrete answer spaces, like math problems with numerical outputs, Pass@k at large ‘k’ values can reflect an increased chance of success due to random guessing rather than genuine reasoning ability. If a model has even a tiny, non-zero probability of getting the correct answer, given enough attempts, Pass@k will eventually approach 100%. This doesn’t necessarily mean the model ‘reasons’ better; it just means it got lucky over many trials.
Introducing Cover@τ: A New Perspective on Reliability
To address these limitations, the researchers propose a new metric called Cover@τ. This metric measures the fraction of problems a model can solve for which at least a ‘τ’ proportion of its generated completions are correct. Unlike Pass@k, Cover@τ introduces an explicit reliability threshold. Models that rely heavily on random chance will see their Cover@τ scores degrade rapidly as ‘τ’ (the required proportion of correct completions) increases.
Cover@τ offers a more nuanced view of an LLM’s capabilities, revealing a ‘breadth-depth trade-off’. At low ‘τ’ values, it captures the ‘breadth’ of problem-solving – how many problems a model can solve, even if inconsistently. At higher ‘τ’ values, it indicates ‘depth’ – how reliably a model can solve problems. For instance, a model might solve many problems occasionally (high breadth, low depth), while another solves fewer problems but almost always correctly (low breadth, high depth). Cover@τ allows us to differentiate between these scenarios.
The Relationship Between Pass@k and Cover@τ
The paper mathematically demonstrates that Pass@k is actually a weighted average of Cover@τ. This means Pass@k only summarizes a part of the richer information contained within Cover@τ. Crucially, Pass@k is biased towards low ‘τ’ regions, emphasizing ‘lucky hits’ rather than consistent reliability. As ‘k’ (the number of attempts) grows, Pass@k increasingly focuses on whether a task has *any* non-zero success probability, rather than how reliably it can be solved.
Also Read:
- Accurate Pass@k Prediction for Large Language Models
- Unlocking How Large Language Models Perceive Problem Difficulty
Re-evaluating LLM Performance
By applying Cover@τ to evaluate various RLVR models on math datasets like OMEGA and Reasoning Gym, the authors show how the relative rankings of popular algorithms can change significantly compared to Pass@1 or Pass@k. For example, on out-of-distribution (OOD) tasks, the base model might appear to perform strongly with Pass@k at large ‘k’, but its Cover@τ curve quickly drops even at modest reliability thresholds, suggesting a far more limited reasoning boundary than Pass@k implies. Methods designed to enhance exploration, such as KL-Cov, often show stronger generalization abilities and higher reliability scores under Cover@τ.
In conclusion, Cover@τ provides a more informative and reliable assessment of LLM reasoning capabilities by explicitly accounting for the consistency of correct answers. It helps researchers and developers understand the true ‘reasoning boundaries’ of their models, moving beyond metrics that can be inflated by chance. For more details, you can read the full research paper here.
