TLDR: ARISE is a new metric designed to evaluate the test-time scaling capabilities of large reasoning models. It introduces sample-level awareness to penalize performance degradation and wasteful computation, and a dynamic sampling mechanism to ensure stable and reliable measurements despite inherent model variability. Experiments show ARISE provides a more consistent, discriminative, and stable evaluation compared to traditional methods, highlighting models like Claude Opus 4.1 as top performers in scaling efficiency.
In the rapidly evolving landscape of large reasoning models, a new paradigm called test-time scaling has emerged. This innovative approach allows models to dynamically adjust their computational resources during inference, aiming to boost performance on complex tasks. As more models integrate this capability, a crucial question arises: how can we accurately and systematically compare their test-time scaling effectiveness? A recent research paper introduces a novel metric called ARISE (Adaptive Resolution-aware Scaling Evaluation) to address this very challenge. The paper, titled “ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Models,” was authored by Zhangyue Yin, Qiushi Sun, Zhiyuan Zeng, Zhiyuan Yu, Qipeng Guo, Xuanjing Huang, and Xipeng Qiu.
Traditional methods for evaluating test-time scaling, such as those based on simple slope calculations, have notable limitations. They often operate at an aggregate level, failing to capture individual sample behaviors. This means they might not penalize instances where increased computation actually leads to a degradation in performance – a phenomenon the researchers call “negative scaling.” Imagine a model trying harder but getting worse results; existing metrics might not adequately reflect this inefficiency. Furthermore, the inherent variability in model outputs and token consumption during inference makes stable and reproducible assessments difficult.
ARISE tackles these issues with two key innovations. First, it incorporates “sample-level awareness.” Instead of just looking at overall accuracy, ARISE tracks the journey of each individual problem or “sample” as the model scales its efforts. This allows the metric to effectively penalize situations where a sample that was initially correct becomes incorrect after more computation, or where computational resources are wasted for no performance gain. The penalty for degradation is designed to be stronger than the reward for improvement, reflecting the higher cost of computational waste.
Second, ARISE employs a “dynamic sampling mechanism.” Large reasoning models can be quite stochastic, meaning their outputs and resource usage can vary across different runs. To ensure reliable measurements, ARISE adaptively adjusts the number of evaluation runs for each sample based on how much its accuracy and token consumption fluctuate. If a sample shows high variability, it gets more trials, leading to a more statistically robust assessment. Conversely, stable samples require fewer trials, optimizing computational efficiency.
The researchers conducted extensive experiments across a diverse range of tasks, including mathematical reasoning (AIME, HMMT), scientific problem-solving (GPQA-Diamond, MMLU-Pro), code generation (SWE-bench Verified, LiveCodeBench), agentic tasks (τ2-Bench, BFCL-v3), and even multimodal reasoning (MMMU, MathVista, CharXiv-Reasoning). Their findings highlight ARISE’s effectiveness. For instance, Claude Opus 4.1 consistently achieved the highest ARISE scores, demonstrating superior scaling characteristics. Among open-source models, Qwen3-235B-A22B showed strong test-time scaling capabilities.
Crucially, ARISE successfully identified models exhibiting negative scaling behaviors, such as GPT-OSS-20B and DeepSeek-R1, which showed performance degradation despite increased computation. This critical insight is often missed by traditional metrics that only yield positive scores. ARISE also demonstrated superior stability and consistency across different datasets and tasks, providing a more reliable and fine-grained measurement of scaling efficiency. It even proved sensitive enough to track the progression of scaling capabilities across different generations of models, showing clear improvements in newer versions like Claude Opus 4.1 compared to its predecessors.
Also Read:
- MixReasoning: A Smart Approach to Efficient Language Model Thinking
- Adaptive Sampling Enhances Stability and Efficiency in LLM Reinforcement Learning
In essence, ARISE offers a more principled and robust framework for evaluating how well large reasoning models can dynamically scale their computational efforts. By focusing on individual sample trajectories and adapting to inherent variability, it provides a clearer picture of true scaling effectiveness, helping to guide the development of more efficient and reliable AI systems.
