TLDR: ConfProBench is the first benchmark to evaluate the reliability of step-level confidence scores in Multimodal Large Language Model (MLLM)-based Process Judges (MPJs). It uses adversarial perturbations and introduces new metrics (Robustness, Sensitivity, Calibration) to assess confidence. Experiments on 14 MLLMs reveal limitations in current MPJs’ confidence performance, showing varying impacts of model scale and “thinking” modes, and highlighting areas for future improvement in AI trustworthiness.
In the rapidly evolving landscape of artificial intelligence, Multimodal Large Language Models (MLLMs) are becoming increasingly adept at tackling complex tasks that involve both text and images. A crucial aspect of these models is their ability to reason through problems step-by-step. To ensure the reliability of this reasoning, a new class of AI tools, known as MLLM-based Process Judges (MPJs), has emerged. These MPJs are designed to evaluate the correctness of each step in an MLLM’s reasoning process.
While existing benchmarks for MPJs focus on evaluating their ability to classify step correctness or search reasoning processes, a critical dimension has been largely overlooked: the reliability of the confidence scores these MPJs produce at each step. Confidence scores are vital because they reflect a model’s self-assessed certainty and directly impact the trustworthiness and safety of AI systems in real-world applications. To address this gap, researchers have introduced ConfProBench, the first comprehensive benchmark specifically designed to systematically evaluate the reliability of step-level confidence scores generated by MPJs. You can find the full research paper here: ConfProBench Research Paper.
Understanding ConfProBench’s Approach
ConfProBench introduces a novel approach by constructing three types of adversarially perturbed reasoning steps. These perturbations are designed to test the robustness of an MPJ’s confidence under various changes that preserve the original meaning but alter the input. The three types of perturbations are:
Synonym Substitution: Replacing non-technical terms with synonyms.
Syntactic Transformation: Altering sentence structure while maintaining semantic information.
Image Perturbation: Applying transformations like scaling, rotation, noise, or color inversion to images.
By evaluating MPJs against these perturbed inputs, ConfProBench can determine how stable and reliable their confidence scores are when faced with slight variations.
New Metrics for Confidence Evaluation
To provide a thorough assessment, ConfProBench proposes three new evaluation metrics, each capturing a complementary aspect of confidence performance:
Confidence Robustness Score (CRS): This metric measures how consistently an MPJ’s confidence scores remain unchanged when presented with adversarially perturbed inputs. A higher CRS indicates greater stability.
Confidence Sensitivity Score (CSS): CSS quantifies how well an MPJ’s confidence scores respond to errors. Ideally, confidence should significantly decrease when an error is present, indicating the model is sensitive to incorrect reasoning.
Confidence Calibration Score (CCS): CCS evaluates the alignment between an MPJ’s confidence scores and its actual prediction accuracy. A well-calibrated model’s confidence should accurately reflect the probability of its predictions being correct.
Key Findings from Extensive Experiments
The researchers evaluated 14 state-of-the-art MLLMs, including both proprietary models like GPT-4o and Gemini-2.5-flash, and open-source models such as InternVL3 and Qwen2.5-VL. The experiments revealed several important insights:
Current MPJs show significant limitations in their confidence performance, indicating ample room for improvement.
Surprisingly, some open-source MPJs demonstrated higher confidence robustness (CRS) than certain proprietary models, suggesting that proprietary models are not always superior in this aspect.
Proprietary MPJs generally outperformed open-source models in confidence sensitivity (CSS) and calibration (CCS).
MPJs showed the lowest robustness when facing syntactic transformations, highlighting this as a particular challenge for maintaining consistent confidence.
The impact of model size on confidence performance varies. While confidence sensitivity often improves with scale, robustness can sometimes decrease, and calibration doesn’t always show a consistent upward trend with increasing size.
For models with a “thinking process” (like Gemini-2.5-flash), this feature can enhance confidence robustness and sensitivity, but it doesn’t necessarily improve calibration quality.
Also Read:
- Fostering LLM Teamwork: A Reinforcement Learning Approach to Collaborative AI
- SEAgent: An AI Framework for Autonomous Software Proficiency
Looking Ahead
ConfProBench provides a foundational step towards building more trustworthy and reliable MLLM-based process judges. The findings underscore the need for future research to focus on improving confidence robustness, sensitivity, and calibration, especially under various adversarial conditions. Future work could also involve human confidence annotations to align MPJ confidence with expert judgments and extending the benchmark to safety-critical scenarios where highly reliable confidence estimation is paramount.
