TLDR: OckBench is a new benchmark that evaluates large language models (LLMs) not just on accuracy, but also on their decoding token efficiency for reasoning and coding tasks. It reveals that models with similar accuracy can have vastly different token consumption, impacting latency, cost, and energy. The benchmark highlights a significant efficiency gap between commercial and open-source models and advocates for a shift in evaluation to prioritize token-efficient reasoning.
Large language models (LLMs) like GPT-4, Claude 3, and the Gemini series have significantly advanced automated reasoning and code generation. However, most existing benchmarks primarily focus on accuracy and output quality, often overlooking a crucial aspect: the efficiency of token generation. The difference between generating 10,000 tokens versus 100,000 tokens can have substantial implications for latency, cost, and energy consumption in real-world systems.
To address this critical oversight, researchers have introduced OckBench, the first model-agnostic and hardware-agnostic benchmark designed to measure both accuracy and the decoding token count for reasoning and coding tasks. This new benchmark aims to shift the evaluation paradigm, arguing that tokens should no longer be treated as “free” resources.
The Need for Efficiency Measurement
The principle of Ockham’s Razor, “Entities must not be multiplied beyond necessity,” serves as the inspiration for OckBench. While LLMs excel at complex problem-solving through techniques like Chain of Thought (CoT) prompting, the computational costs associated with these reasoning processes have grown significantly. Public reports indicate that frontier models can take many hours to solve a few mathematical problems or coding challenges. Despite these substantial time and computational costs, the community often prioritizes accuracy, with efficiency receiving far less attention in evaluations like HELM, LM-Eval, and LMSYS Chatbot Arena.
The practical cost of LLMs is a major deployment bottleneck. Each additional decoding token contributes to latency, energy consumption, and monetary cost, with LLM service providers often billing based on output tokens. Empirical analysis also shows that the response lengths of reasoning-capable models are growing much faster than non-reasoning models, highlighting the increasing “hidden cost” of thinking in token form.
OckBench: A Unified Framework
OckBench proposes a new evaluation perspective centered on intrinsic token efficiency. Its key contributions include:
-
Model-Agnostic Efficiency Metric: It formalizes decoding token count as an intrinsic, hardware- and system-independent efficiency metric, providing a more holistic view of model performance alongside accuracy.
-
Efficiency-Accuracy Aware Benchmark: OckBench is the first unified benchmark specifically designed to evaluate an LLM’s reasoning process efficiency by measuring decoding token consumption and accuracy.
-
Empirical Efficiency-Accuracy Trade-offs: Experiments reveal substantial practical trade-offs, illustrating how models distribute on an accuracy–efficiency Pareto frontier.
Benchmark Composition and Methodology
OckBench is structured to test LLMs’ reasoning efficiency across two main domains: mathematics problem-solving and coding skills. For mathematics, it uses challenging problems from GSM8K, AIME24, and AIME25. For coding, it employs a variant of MBPP and 200 curated real-world coding problems. A core methodological choice for OckBench is selecting the top 200 questions that exhibit high variance in decoding token usage among baseline models. This ensures the benchmark focuses on instances where token count is a decisive metric, revealing true reasoning efficiency and accuracy-efficiency trade-offs.
Experimental Findings
The benchmark evaluated a range of open- and closed-source models. The experiments uncovered a significant reasoning efficiency gap between commercial (closed-source) and open-source models.
Commercial Models: These models demonstrated superior overall performance, with an average accuracy of 60.8%. GPT-5 achieved the highest accuracy at 73%. However, there was wide variance in token efficiency; GPT-5 was accurate and concise (2,336 tokens), while Gemini-2.5 Pro required over twice as many tokens (5,198) for slightly lower accuracy. GPT-4o emerged as the most token-efficient commercial model, though with lower accuracy than top performers.
Open-Source Models: These models had a lower average accuracy of 35.3%. NVIDIA’s AceReason-Nemotron-14B and Qwen’s Qwen3-14B were top performers in this category (40% accuracy). A clear trend showed that “thinking” variants of Qwen models, likely using more extensive chain-of-thought processing, produced substantially higher token counts without a proportional accuracy increase. NovaSky-AI Sky-T1-7B offered a good balance of performance and efficiency within the open-source group, achieving respectable accuracy with a low average token count, comparable to efficient commercial models.
Also Read:
- Evaluating AI’s Coding Prowess in Ukrainian: Introducing UA-Code-Bench
- New Benchmark Reveals AI Agents Struggle with Real-World Software Performance Optimization
Conclusion
OckBench highlights that token efficiency is a meaningful differentiator, especially in deployment scenarios where latency, computation, and cost are critical. By providing a model- and hardware-agnostic metric, OckBench offers a reproducible platform for comparing the accuracy–efficiency trade-off of reasoning models. The hope is that this benchmark will guide the community towards designing models that are not only accurate but also employ leaner, more efficient reasoning. For more details, you can read the full research paper here.
