TLDR: A study found that while visualizations increase MLLM confidence in network tasks like bridge detection, they also introduce strong biases, causing MLLMs to make decisions based on the visualization style rather than the actual data, potentially leading to confident but incorrect “hallucinations.”
Recent advancements in generative AI have led to the increasing use of large language models (LLMs) as decision-makers across various fields like law, finance, and healthcare. With the emergence of multi-modal large language models (MLLMs), which can process different types of input, there’s a growing interest in how visualizations might influence their decision-making, similar to how humans use visual analytics.
A new research paper, “Visualization Biases MLLM’s Decision Making in Network Data Tasks,” explores this very topic. The study, presented at VIS x GenAI, a workshop co-located with IEEE VIS 2025, investigates how different visualization techniques can influence MLLMs’ judgments, specifically concerning the presence or absence of “bridges” in a network. A bridge is a single edge whose removal would disconnect the network. If a network has no bridge, it is considered 2-edge-connected.
The researchers, Timo Brand, Henry F ¨orster, Stephen G. Kobourov, and Jacob Miller, aimed to answer two key questions: Can the accuracy and confidence of MLLMs be improved with suitable visualizations, and does visualization introduce bias, depending on the style used?
The experimental setup involved presenting MLLMs (GPT-4o and Qwen2.5-vl-72b-instruct) with small to medium-sized networks. These networks were described in text form by an adjacency matrix, and in some cases, additional visual information was provided. The configurations included text-only inputs (unstructured and structured adjacency matrices) and inputs with visualizations (pixel-based adjacency matrix, circular layout, and force-directed spring layout).
The findings were quite striking. While the inclusion of visualization did improve confidence over structured text-based input, it also created a strong bias. Standard visualization techniques consistently steered the MLLM’s decision towards either accepting or refuting the presence of a bridge, regardless of whether a bridge actually existed in the network. For example, the pixel-based adjacency matrix (AM) and spring layouts often led MLLMs to report a bridge, even when none was present. Conversely, the circular layout increased the probability of the MLLM indicating a 2-edge-connected network, even for networks that did contain a bridge.
Interestingly, both MLLMs reported very high confidence scores (often the highest possible, 5 out of 5) in a large majority of trials, irrespective of whether their answer was correct or incorrect, or whether the instance was positive (had a bridge) or negative (no bridge). This suggests that while visualizations can boost an MLLM’s confidence, they don’t necessarily improve accuracy and can lead to “hallucinations” or incorrect judgments.
The study highlights that MLLMs are significantly influenced by visualization design, sometimes more so than by the underlying data itself. This effect, while seemingly parallel to human perception, shows critical differences. For instance, MLLMs did not perceive significant differences in illustration pairs that a human likely would. This raises important considerations for practitioners using generative AI applications, as allowing users to include visualizations could lead to undesired outcomes and biased decision-making.
Also Read:
- Unpacking How AI Models Resolve Conflicting Information from Images and Text
- Unpacking Multimodal AI Failures: How One Data Stream Can Undermine Others
In conclusion, while visualizations can make MLLMs more confident, they also introduce strong biases that are dependent on the visualization style. This research underscores the need for caution when designing applications that use visualizations to guide MLLM judgments, emphasizing that evaluating MLLMs as “human-like” readers can be problematic. For more details, you can read the full paper here.
