TLDR: A new research paper reveals that Vision Language Models (VLMs) often fail at seemingly simple visual reasoning tasks because they lack ‘visually-grounded serial processing’—the human ability to break down complex visual problems into sequential steps. The study compared human and VLM performance across geometric reasoning, numerical estimation, and mental rotation tasks, finding that VLM accuracy consistently declined as human reaction times (a proxy for serial processing load) increased. While augmenting VLMs with serial processing methods showed some improvements, the findings suggest a fundamental bottleneck in current AI architectures, highlighting the need for new approaches to achieve more human-like visual intelligence.
Vision Language Models (VLMs) have shown remarkable capabilities in tasks like image captioning and visual question answering, often matching or even surpassing human performance on complex benchmarks. However, these advanced AI models sometimes struggle with surprisingly simple visual reasoning tasks, such as counting or basic spatial understanding. A recent research paper titled “Visual serial processing deficits explain divergences in human and VLM reasoning” proposes a compelling explanation for this paradox: a fundamental deficit in visually-grounded serial processing in VLMs. You can read the full paper here: Visual serial processing deficits explain divergences in human and VLM reasoning.
The researchers, Nicholas Budny, Kia Ghods, Declan Campbell, Raja Marjieh, Amogh Joshi, Sreejan Kumar, Jonathan D. Cohen, Taylor W. Webb, and Thomas L. Griffiths, hypothesize that while humans can adapt to visual complexity by taking more time and using sequential attention to analyze different elements in a scene, current VLMs largely lack this ability. They suggest that VLM reasoning is tied to generating text sequentially (their ‘chain of thought’) rather than sequentially analyzing the image itself. To test this, they compared human and VLM performance across tasks designed to vary the demands on serial processing in three distinct areas: geometric reasoning, perceptual enumeration (counting), and mental rotation.
Geometric Reasoning: Complexity and Performance
In the geometric reasoning task, participants and models had to identify an ‘oddball’ image that violated a specific geometric concept within an array of six images. The complexity of these concepts was varied using a measure called Minimum Description Length (MDL), which represents the number of geometric primitives needed to define a concept. The study found a clear pattern: as the geometric concept complexity (MDL) increased, human reaction times (the time taken to respond) also increased, indicating more cognitive effort. In contrast, VLM accuracy declined significantly under these same conditions. This suggests that while humans can compensate for complexity with slower, sequential reasoning, VLMs struggle to handle deeper compositional programs.
Numerical Estimation: Counting Challenges
The second domain explored was numerical estimation, or counting objects in a scene. Humans typically use ‘subitizing’ for small quantities (up to 4-6 items), recognizing them almost instantly. For larger quantities or when objects are harder to distinguish, humans switch to serial enumeration, deploying sequential attention to identify each item. The researchers manipulated numerosity (1-8 objects), spatial arrangement (overlapping vs. non-overlapping), and color (uniformly vs. uniquely colored) to vary the perceptual individuation load. Humans maintained accuracy across conditions, albeit with increased reaction times in more challenging scenarios (like overlapping objects). VLMs, however, showed a significant drop in accuracy in overlapping conditions and struggled with higher numerosities. Interestingly, VLMs performed better when overlapping objects were uniquely colored compared to uniformly colored. This suggests that distinct colors allowed VLMs to individually reference and count objects, grounding their reasoning in linguistic descriptions, a benefit not observed in humans.
Mental Rotation: Spatial Transformations
Mental rotation provided a particularly stringent test, as it involves manipulating spatial representations through continuous transformations, which are difficult to express in discrete, language-based tokens. In this task, participants and models judged whether two rotated shapes were identical or mirror-reversed. Decades of cognitive science research show that human reaction time increases linearly with the angular disparity between the objects, reflecting an analog, incremental transformation process. While human accuracy declined modestly with increased rotation angle, VLM accuracy dropped sharply, especially beyond 75 degrees. This strong inverse correlation between human reaction time and VLM accuracy highlights that tasks requiring deeper, stepwise transformations in humans expose fundamental limitations in current VLMs, which appear unable to perform continuous internal transformations.
Augmenting VLMs: A Glimpse of Serial Processing
To further investigate the serial processing deficit, the researchers augmented VLMs with forms of serial processing like Chain-of-Thought (CoT), reasoning training, and tool use (e.g., cropping and rotating images). These augmentations showed improvements in specific tasks. For instance, CoT and reasoning training helped in numerosity tasks where objects could be uniquely referenced by color. Tool use significantly improved performance in the rotation task and in numerosity tasks with spatially separated objects. These results provide causal evidence that enabling serial processing can help VLMs solve complex visual tasks. However, they also reveal the limitations of current augmentation methods, as they are effective only in specific settings, pointing to the need for more intrinsically visual forms of serial processing.
Also Read:
- UniAPL: Unifying Language Model Training for Enhanced Instruction Following
- Unmasking AI Judges: A New Approach to Detecting LLM-Generated Evaluations
Conclusion: A Unifying Explanation and Future Directions
The study concludes that the serial processing deficit offers a unifying explanation for why VLMs, despite their impressive general capabilities, falter on seemingly simple visual reasoning tasks. The consistent inverse correlation between VLM performance and human reaction time across diverse domains strongly supports this hypothesis. This understanding not only sheds light on current model limitations but also provides a roadmap for future architectural innovations. Developing models with visually grounded reinforcement learning or sequential attention mechanisms that mimic human saccade-and-fixate cycles could be key to achieving more human-like flexibility and robustness in visual reasoning, combining the parallel pattern recognition strengths of current VLMs with the crucial serial processing characteristics of human cognition.
