TLDR: This research introduces JBA, a new, comprehensive, and automatically generated benchmark dataset to evaluate how well Multimodal Large Language Models (MLLMs) can identify false premises in questions. It categorizes false premises into 13 subtypes across three levels (Perceptual, Cognitive, Reasoning). The paper also proposes JBA-GRPO, a reinforcement learning framework with a novel “reasoning reward,” designed to improve MLLMs’ ability to detect and reject false premises. Experiments show that current MLLMs struggle with this task, but models trained with JBA-GRPO achieve significant improvements, making MLLMs more reliable.
Multimodal Large Language Models (MLLMs) have made incredible strides in understanding and generating content that combines both vision and language. However, despite their advanced capabilities, a significant challenge persists: the “false premise problem.” This occurs when an MLLM is given a question that contains factually incorrect or nonsensical information. Ideally, the model should identify and reject this invalid premise. Unfortunately, current MLLMs often accept these false assumptions, leading them down a path of flawed reasoning and ultimately providing confidently delivered, yet unreliable, answers.
Existing methods and benchmarks designed to address this issue have been limited. They often lack detailed categorization of false premises, don’t cover enough scenarios, and therefore fail to rigorously evaluate a model’s true ability to recognize these errors. This creates a significant gap in both evaluation and the development of targeted training methods.
Introducing JBA: A Comprehensive Benchmark for False Premises
To tackle this evaluation gap, researchers have introduced JBA, a comprehensive dataset specifically designed for false premise questions. What makes JBA unique is its fully automated construction pipeline. This innovative approach systematically categorizes premises into a hierarchical taxonomy of three main levels—Perceptual, Cognitive, and Reasoning—and further breaks them down into thirteen distinct subtypes. This detailed categorization allows for a much more precise analysis of where MLLMs struggle, from basic visual perception to complex abstract reasoning.
The automated pipeline works in three stages: First, a Multimodal Large Language Model extracts a specific premise from an input image. Second, it generates a concise caption for the image that explicitly includes this extracted premise. Finally, a Large Language Model creates both positive (true premise) and negative (false premise) question samples by either embedding the correct premise or replacing it with an incorrect one. This systematic generation process ensures a large-scale, diverse, and finely categorized dataset, significantly reducing the need for manual annotation.
JBA-GRPO: Enhancing MLLM Robustness
Beyond just evaluation, the research also addresses the methodological gap by proposing JBA-GRPO, a reinforcement learning framework tailored to strengthen MLLMs’ ability to detect false premises. JBA-GRPO builds upon the Group Relative Policy Optimization (GRPO) strategy, which is known for enhancing the deliberative capabilities of large language models.
A key innovation in JBA-GRPO is its unique “reasoning reward.” While traditional rewards focus on correct formatting and final answer accuracy, they don’t always guarantee logical integrity in the reasoning process itself. The reasoning reward uses a separate Large Language Model as an evaluator to assess the quality and coherence of the MLLM’s internal thought process (captured in a special <think> block). High-quality, logically sound reasoning receives a positive reward, while flawed reasoning incurs a penalty, ensuring that models learn to arrive at correct answers through valid steps.
The training strategy for JBA-GRPO involves a two-stage process: an initial Supervised Fine-Tuning (SFT) phase to align the model with the required reasoning structure, followed by a Reinforcement Learning (RL) optimization phase using the custom reward functions to refine advanced reasoning abilities.
Also Read:
- Video-STR: Enhancing AI’s Understanding of Object Relationships and Motion in Videos
- OmniVideoBench: A New Benchmark for Advanced Audio-Visual AI Understanding
Experimental Findings and Future Outlook
Experiments conducted on the JBA dataset with several representative MLLMs, including InternVL3, Qwen2.5-VL, LLaVA-v1.5-7B, and LLaVA-OneVision, revealed that current models still struggle significantly with false premise recognition. However, models trained with the JBA-GRPO framework consistently achieved substantial improvements across all evaluation metrics (False Premise Coverage, False Premise Detection Precision, and True Premise Identification Rate).
The superior performance of the JBA model is attributed to two main design principles: the structured reasoning mechanism enforced by the <think> tag, which prompts the model to explicitly analyze visual and textual content for discrepancies, and the reinforcement learning framework that refines and generalizes this reasoning ability. This work establishes a new standard for reliable multimodal reasoning.
In conclusion, this research not only provides a much-needed comprehensive benchmark for evaluating false premise detection in MLLMs but also offers an effective training framework to enhance their robustness. These findings underscore the critical need for specialized training to improve MLLMs’ ability to detect and manage false premises across diverse scenarios, paving the way for more trustworthy and reliable AI systems. The code for this research is publicly available, allowing others to build upon these advancements. You can find more details in the full research paper: JUDGE BEFORE ANSWER: CAN MLLM DISCERN THE FALSE PREMISE IN QUESTION?
