TLDR: GRAFT is a new benchmark designed to evaluate multimodal AI models on their ability to understand and reason with structured visual data like charts and tables. It uses programmatically generated synthetic data and multi-step analytical questions, requiring answers in structured formats like JSON or YAML. The benchmark assesses various reasoning types and highlights current models’ struggles with correctness and visual grounding, especially for complex graphics, despite good format fidelity.
In the rapidly evolving landscape of artificial intelligence, multimodal foundation models are making significant strides, excelling in tasks that combine different types of data, such as images and text. However, a crucial gap remains in how well these models handle structured visual data like charts and tables, especially when complex reasoning and precise, structured outputs are required. To address this, researchers from ServiceNow have introduced GRAFT: GRaph And Table Reasoning for Textual Alignment – a new benchmark designed to rigorously evaluate AI models on these challenging tasks.
GRAFT stands out as a structured multimodal benchmark specifically crafted for assessing instruction-following, visual reasoning, and visual-textual alignment. Unlike many existing benchmarks that focus on natural images or scanned documents, GRAFT emphasizes structural complexity and controllability. It achieves this by featuring programmatically generated charts and synthetically rendered tables, created using Python visualization libraries. This programmatic approach ensures precise control over data semantics, structure, and visual clarity, making it an ideal environment for testing AI capabilities.
Each instance within the GRAFT benchmark pairs a chart or table image with a systematically generated, multi-step analytical question. These questions are designed to be answerable solely based on the visual content, preventing models from relying on external knowledge. The answers are provided in structured formats, such as JSON or YAML, which allows for consistent and objective evaluation of both the model’s reasoning process and its ability to generate outputs in a specified format.
The benchmark introduces a comprehensive taxonomy of reasoning types to enable a fine-grained assessment of AI models. These include critical analytical skills such as comparison, trend identification, ranking, aggregation, proportion estimation, and anomaly detection. By covering such a wide range of reasoning tasks, GRAFT provides a unified and scalable framework for benchmarking multimodal models on visually grounded, structured reasoning, setting a new standard in the field.
The methodology behind GRAFT involves a sophisticated data generation and validation process. For charts, tabular data (typically 8–20 rows and 3–6 columns) is visualized as bar, line, pie, or scatter plots using popular Python libraries like Matplotlib and Seaborn. Similarly, tables are synthesized for various formats (e.g., pivot, grouped, sortable) and rendered as stylized images. After visual generation, a crucial “Jury-of-Judges” filtering step, involving multiple large language models (LLMs) as independent evaluators, scores answers for correctness, completeness, visual grounding, and format fidelity, ensuring high-quality evaluation data.
GRAFT is divided into two main subsets for evaluation: Chart-QnA, which involves reasoning over various chart types, and Table-QnA, focusing on analytical tasks with realistic, domain-specific tables. The evaluation pipeline uses a two-stage process: first, a Vision-Language Model (VLM) generates YAML/JSON answers, and then a GPT4o-based automated judge scores these answers on a scale of 1 to 5 across four axes: Correctness, Completeness, Visual Grounding, and Format Fidelity.
Initial experimental results on GRAFT reveal interesting insights into current VLM performance. While models like Qwen-2.5 32B VL generally lead in performance, especially in correctness, many models, including GPT4o, Pixtral 12B, and Mistral 24B, demonstrate strong capabilities in format fidelity and completeness. However, they often struggle with correctness and visual grounding, particularly when faced with complex graphics or multi-step reasoning tasks over tables. This highlights that while models can be guided to produce well-structured outputs, deep visual-semantic parsing and accurate reasoning remain significant challenges.
Also Read:
- Assessing AI’s Visual Smarts: Introducing MM-BrowseComp for Multimodal Web Browsing
- Assessing AI’s Spatial Awareness: A Deep Dive into Image Rotation Challenges
The researchers acknowledge certain limitations, such as the reliance on synthetic data, which might impact real-world generalizability, and the current restriction to English language coverage. Future work aims to incorporate human-annotated benchmarks, real-world and multilingual data, and richer task formats like multi-hop reasoning to further enhance GRAFT’s robustness and utility. This benchmark provides a robust platform for evaluating and advancing future multimodal QA models, pushing the boundaries of what AI can understand from visual data. You can read the full research paper here.
