TLDR: This survey paper reviews the rapid evolution of Compositional Visual Reasoning (CVR) from 2023 to 2025, analyzing over 260 papers. It defines CVR, explains its advantages over monolithic AI models, and outlines five key developmental stages: from language-centric prompting to unified agentic vision-language models. The paper also catalogs benchmarks, identifies challenges like hallucinations and data limitations, and proposes future research directions for building more human-like, interpretable, and robust visual reasoning systems.
Artificial intelligence is constantly striving to mimic human-like abilities, and one of the most fascinating areas of research is how machines interpret and understand the visual world. A recent survey, titled “Explain Before You Answer: A Survey on Compositional Visual Reasoning,” delves into the rapid advancements in Compositional Visual Reasoning (CVR), a field aiming to give AI the ability to break down complex visual scenes, understand individual concepts, and perform multi-step logical thinking, much like humans do.
Authored by Fucai Ke, Joy Hsu, Zhixi Cai, Zixian Ma, Xin Zheng, Xindi Wu, Sukai Huang, Weiqing Wang, Pari Delir Haghighi, Gholamreza Haffari, Ranjay Krishna, Jiajun Wu, and Hamid Rezatofighi, this comprehensive survey reviews over 260 papers published between 2023 and 2025. It highlights a significant shift from traditional ‘monolithic’ AI models, which act as black boxes, directly mapping visual and text inputs to answers without showing their work.
Why Compositional Visual Reasoning Matters
The paper explains that monolithic models often struggle with complex visual tasks because they rely on superficial patterns and can produce incorrect answers, a phenomenon known as ‘hallucination.’ They also face challenges with tasks requiring multi-step reasoning or precise understanding of spatial relationships. CVR, on the other hand, offers several key advantages:
- Cognitive Alignment: It mirrors how humans naturally break down scenes into objects, attributes, and relationships.
- Semantic Understanding: It explicitly models relationships, leading to deeper comprehension.
- Generalization: It can solve new tasks by recombining familiar elements, reducing reliance on specific training data biases.
- Transparency and Interpretability: By showing intermediate steps (like identifying objects or their relationships), CVR makes AI decisions more understandable and debuggable.
- Reduced Hallucinations: Explicitly grounding reasoning steps in visual evidence helps prevent the model from generating plausible but incorrect information.
- Data Efficiency: Reusing learned visual skills across tasks reduces the need for massive, new datasets.
The Evolution of Visual Reasoning Paradigms
The survey traces CVR’s evolution through five distinct stages:
Stage I: Prompt-Enhanced Language-Centric Methods
Early approaches used Large Language Models (LLMs) to break down complex visual questions into simpler sub-questions. These sub-questions were then answered by Vision-Language Models (VLMs), with the LLM synthesizing the final answer purely in the language space. This modular design improved interpretability but had limited visual grounding.
Stage II: Tool-Enhanced Large Language Models
This stage equipped LLMs with the ability to call external tools (like object detectors or captioners) for perception and analysis. The LLM acted as a central planner, generating tool actions and interpreting results. While more flexible, these systems still relied on textual descriptions of visual information, which could lead to information loss.
Stage III: Tool-Enhanced Vision-Language Models
A crucial advancement, this stage replaced the LLM planner with a VLM, allowing direct access to raw images. This reduced information loss and enabled more accurate planning. These VLMs could even use external tools to generate or modify images, simulating ‘visual imagination’ and verification.
Stage IV: Chain-of-Thought Reasoning VLMs
Moving towards more integrated systems, these VLMs perform multi-step reasoning without external tools. They explicitly reveal intermediate ‘thought’ processes and perception information before providing a final answer, often through structured prompting or reinforcement learning.
Stage V: Unified Agentic Vision-Language Models
The latest evolution, these models are designed to be autonomous. They actively plan, adapt, ‘imagine,’ and execute sequences of decisions to solve complex visual tasks. They can automatically discover informative regions in an image, explore them, and even simulate mental imagery internally to refine their reasoning. This stage represents a significant step towards truly intelligent visual agents.
Also Read:
- Unifying Visual Perception: A Deep Dive into Open World Detection
- Enhancing Multi-Image Question Answering in AI Models with Adaptive Visual Anchoring
Challenges and Future Directions
Despite rapid progress, CVR faces several hurdles. LLMs, while powerful, often lack an internal ‘world model’ to simulate physical dynamics or spatial transformations, limiting their ability to reason about hypothetical scenarios. Hallucinations, though reduced, can still occur if visual grounding is insufficient. Most current systems also lean heavily on ‘deductive reasoning,’ which can be brittle if initial information is flawed, suggesting a need for ‘inductive’ (generalizing from observations) and ‘abductive’ (generating plausible explanations) reasoning.
Data scarcity is another major challenge, as high-quality, step-by-step annotations are expensive. Tool integration also presents architectural bottlenecks, with issues like tool awareness and computational costs. Finally, current benchmarks often fall short, primarily evaluating final answers rather than the quality of intermediate reasoning steps, and lacking clear difficulty levels.
Future research aims to integrate explicit world models into AI systems, allowing them to simulate visual scenarios and plan actions more effectively. Human-in-the-loop supervision can enhance reliability, while developing hybrid data engines that combine synthetic and real-world imagery can address data limitations. The goal is to create more integrated architectures and nuanced evaluation protocols to foster the development of robust, general-purpose, and trustworthy compositional visual reasoning systems. For more details, you can read the full paper here.
