TLDR: CHARTHAL is a new benchmark that evaluates how Large Vision-Language Models (LVLMs) hallucinate when understanding charts. It uses a fine-grained taxonomy of question types and chart-question relations, with a human-validated dataset of 1,062 samples. Experiments show that state-of-the-art LVLMs, including GPT-5 and o4-mini, perform poorly on unanswerable questions (irrelevant, inexistent, or contradictory information), often fabricating answers instead of recognizing unanswerability. This highlights a critical need for better hallucination mitigation strategies in chart understanding.
Large Vision-Language Models (LVLMs) have shown impressive capabilities in understanding and reasoning with visual information, leading to many applications that convert images to text. A particularly valuable yet challenging area for these models is chart understanding, as charts often contain complex information and relationships that require precise visual perception and significant cognitive effort from LVLMs.
Despite their advancements, LVLMs are prone to “hallucination,” which means they generate information that isn’t real or is inconsistent with the input. This problem is especially critical and harmful in chart understanding, where factual accuracy and strict consistency with the visual data are essential. While previous research has looked into hallucinations and chart comprehension separately, the intersection of these two areas has largely been unexplored.
To address this gap, researchers have introduced CHARTHAL, a new benchmark designed to evaluate hallucinations in LVLMs specifically when they interpret charts. CHARTHAL is unique because it offers a detailed classification of hallucination scenarios in chart understanding. It includes a dataset of 1,062 samples that have been validated by humans. These samples cover a variety of chart types and academic fields.
The CHARTHAL framework uses a chart question-answering format, where LVLMs are given a chart image and a question, and then asked to generate an answer. Unlike benchmarks that use simple Yes/No or multiple-choice questions, CHARTHAL uses a free-form answering format to better reflect how LVLMs hallucinate in real-world chart understanding situations. The benchmark categorizes questions along two main dimensions: Question Type and Chart-Question Relation.
Question Types include:
- Descriptive questions: These can be answered by directly extracting information from the chart.
- Reasoning questions: These require calculations or logical deductions based on the chart’s content.
- Open-ended questions: These involve analytical or predictive tasks that don’t have a single definitive correct answer.
Also Read:
- NUMINA: A New Benchmark Reveals AI’s Struggle with 3D Numerical Reasoning
- DivLogicEval: A New Benchmark Uncovers Gaps in LLM Logical Reasoning
Chart-Question Relations categorize questions as:
- Irrelevant: Questions completely unrelated to the chart’s content.
- Inexistent: Questions asking about information that is missing from the chart.
- Contradictory: Questions based on false statements that are inconsistent with the chart’s data.
- Normal: Valid questions that align with the chart’s content.
Questions falling into the “irrelevant,” “inexistent,” and “contradictory” categories are considered “unanswerable” because they cannot be answered based on the provided chart. These types of questions are more likely to trigger hallucinations, as LVLMs tend to produce answers regardless of whether a question can actually be answered.
The evaluation conducted with CHARTHAL involved 15 state-of-the-art LVLMs, including proprietary models like GPT-5 and o4-mini, as well as open-source models such as the Llama 3.2-11B-Vision-Instruct and Qwen2.5-VL series. The results showed that these models suffer from significant hallucinations when interpreting charts. For instance, proprietary models like GPT-5 and o4-mini achieved overall accuracies of only 34.46% and 22.79%, respectively.
Further analysis revealed that questions involving information absent from or contradictory to the charts were particularly likely to cause hallucinations. In these “unanswerable” cases, models frequently made up content instead of correctly identifying that the questions could not be answered. This highlights a critical vulnerability in current LVLMs and emphasizes the urgent need for more robust strategies to reduce hallucinations in chart understanding.
The study also found that larger models generally performed better, suggesting that increasing model scale can improve robustness against hallucinations. However, increasing “reasoning effort” in models yielded only small and inconsistent improvements, and sometimes even worsened performance. This is because if the visual input is misinterpreted, additional reasoning can amplify the error rather than correct it.
In conclusion, while LVLMs can handle standard chart questions reasonably well, they are highly susceptible to hallucination in scenarios where questions are unanswerable. This poses significant risks and underscores the importance of developing advanced techniques, such as adversarial training, to ensure that LVLM-based chart analysis is trustworthy. For more details, you can refer to the full research paper here.
