TLDR: INTER CHART is a new benchmark for evaluating how well AI models (Vision-Language Models) can understand and reason across multiple related charts, a common task in real-world data analysis. It features three tiers of increasing difficulty, from simple decomposed charts to complex real-world multi-domain charts, and uses an LLM-assisted evaluation for more accurate scoring. Initial results show current models struggle significantly with integrating information across complex, diverse charts, highlighting areas for future AI development.
In today’s data-driven world, information is often presented through charts and visualizations. While individual charts can convey specific insights, real-world scenarios like scientific reports, financial analyses, and public policy dashboards frequently require understanding information spread across multiple related charts. This is where a new diagnostic benchmark called INTER CHART comes into play, aiming to evaluate how well artificial intelligence models, specifically vision-language models (VLMs), can reason across these complex, multi-chart environments.
Traditional benchmarks for VLMs often focus on single, isolated charts. However, INTER CHART recognizes that true understanding often emerges from comparing and synthesizing information from several visualizations, even if they differ in style or semantic framing. This new benchmark challenges models with a variety of question types, from simple fact extraction and trend correlation to more complex numerical estimation and abstract multi-step reasoning, all grounded in two to three thematically or structurally related charts.
Three Tiers of Complexity
The INTER CHART benchmark is structured into three tiers, each designed to test different levels of reasoning difficulty. The first tier, named DECAF (Decomposed Elementary Charts with Answerable Facts), focuses on foundational chart understanding. It uses simplified, single-variable charts, often decomposed from more complex figures, to assess direct factual and comparative reasoning in clear visual contexts. This tier helps establish a baseline for how well models can interpret basic visual elements.
The second tier, SPECTRA (Synthetic Plots for Event-based Correlated Trend Reasoning and Analysis), steps up the complexity by introducing synthetic chart pairs. These charts share a common axis but can vary in style, simulating real-world scenarios where relationships between variables evolve over time or across regions. SPECTRA tests a model’s ability to integrate distributed information and reason about correlated trends and event-based interpretations.
The most advanced tier is STORM (Sequential Temporal Reasoning Over Real-world Multi-domain charts). This tier pushes the boundaries of current VLM capabilities by using visually complex, real-world line chart pairs from diverse domains like economic reports and environmental trends. STORM requires models to perform multi-step inference, align mismatched semantics, and synthesize information across different domains and temporal sequences, reflecting the true challenges of real-world data analysis.
A Smarter Evaluation Approach
A key innovation of INTER CHART is its novel evaluation pipeline. Instead of relying on simple exact string matching for answers, the benchmark employs multiple large language models (LLMs) as “semantic judges.” These LLMs assess the correctness of model answers, allowing for flexibility in paraphrased responses, numerical approximations, and equivalent units. This LLM-assisted approach provides a more robust and human-aligned assessment of a VLM’s understanding.
Also Read:
- New Benchmark Reveals AI’s Struggle with Causal Reasoning in Infographics
- Assessing Multimodal AI’s Counting Abilities in Real-World Scenarios
Key Findings and Future Directions
Initial evaluations using INTER CHART have revealed significant insights. State-of-the-art open- and closed-source VLMs show a consistent and steep decline in accuracy as chart complexity increases, particularly in the STORM subset. This suggests that while models perform reasonably well on simplified visuals, they struggle significantly with cross-chart integration and complex real-world scenarios. Interestingly, decomposing multi-entity charts into simpler visual units often improved model performance, underscoring their current limitations in handling integrated, complex visual information.
The research highlights that models like Google Gemini 1.5 Pro generally outperform others across all subsets and prompting strategies, likely due to its strong instruction-following capabilities and training on structured inputs. Open-source models, while performing well on simpler DECAF charts, show a sharp decline in accuracy on the more challenging SPECTRA and STORM subsets, indicating difficulties in generalizing to real-world visual diversity and temporal reasoning.
The findings from INTER CHART provide a rigorous framework for advancing multimodal reasoning in complex, multi-visual environments. By exposing systematic limitations in current VLMs, this benchmark paves the way for future research and development in creating more capable AI systems that can truly understand and reason across the rich tapestry of real-world data visualizations. You can read the full research paper for more details at this link.
