TLDR: A new research paper introduces ReasonBench, a benchmark with 1,613 questions from real intelligence tests, to evaluate Visual Language Models (VLMs) on complex graphic reasoning. The study found that current VLMs achieve only about 25-27% accuracy, significantly lagging behind human performance (68.7%). To improve this, researchers developed a dual optimization framework, combining Diagrammatic Chain-of-Thought (DiaCoT) for better interpretability and ReasonTune for task adaptability, which collectively boosted VLM accuracy by 33.5%.
Visual Language Models, or VLMs, are at the forefront of artificial intelligence, enabling machines to understand and reason using both visual and textual information. These models have shown impressive progress in tasks like describing images and answering questions about visual content. However, a recent study highlights a significant challenge: VLMs still struggle with complex graphic reasoning, a skill crucial for tasks that require abstract problem-solving, similar to human intelligence tests.
Researchers from Beijing Electronic Science & Technology Institute have introduced a new benchmark called ReasonBench to specifically evaluate how well VLMs perform on these intricate graphic reasoning tasks. Unlike previous benchmarks that focused on simpler graphics, ReasonBench includes 1,613 questions drawn from real-world intelligence tests. These questions cover a wide range of reasoning dimensions, including location, attributes, quantity, and multi-element tasks, providing a thorough assessment of spatial, relational, and abstract reasoning abilities in VLMs.
The findings from benchmarking 11 leading VLMs (both commercial and open-source) on ReasonBench reveal a substantial gap between AI and human capabilities. The best-performing VLM achieved only 27% accuracy, while the average across all tested models hovered around 25%. This is a striking result, especially when compared to the human baseline, where participants achieved an average accuracy of 68.7% on the same test set. This stark difference indicates that current VLMs are not yet reliable for independent reasoning and accurate problem-solving in complex graphical scenarios.
The study also explored the impact of different input strategies, such as presenting problems and answer choices in a single integrated graphic versus splitting them into multiple separate graphics. The results showed minimal difference in VLM performance between these two approaches, suggesting that the format of presentation doesn’t significantly alleviate the core reasoning challenges.
To address these limitations, the researchers proposed a dual optimization strategy. The first part is called Diagrammatic Chain-of-Thought (DiaCoT), which aims to improve the interpretability of reasoning by breaking down complex diagrams into layers. This method helps VLMs to explain their thought process step-by-step. The second part is ReasonTune, a fine-tuning strategy designed to enhance the model’s adaptability to specific reasoning tasks through targeted training.
When these two methods were combined in a dual optimization framework, they significantly boosted VLM performance. The experimental results demonstrated an impressive 33.5% improvement in overall reasoning accuracy. This advancement shows promise for developing VLMs that are more capable of handling complex graphic reasoning, moving them closer to human-level performance.
Also Read:
- Improving AI Explanations: CoRGI Introduces Visual Grounding to Chain-of-Thought
- Navigating the Mathematical Landscape: LLMs in Formal and Informal Reasoning
This research, detailed in the paper Oedipus and the Sphinx: Benchmarking and Improving Visual Language Models for Complex Graphic Reasoning, lays a crucial foundation for future developments in VLM technology. While there are still limitations, such as the need for extensive data for training and validation, the ReasonBench benchmark and the proposed optimization framework offer valuable tools for understanding and enhancing the graphic reasoning abilities of AI models.
