TLDR: A new research paper introduces the Turing Eye Test (TET), a benchmark with four perception-oriented tasks (HiddenText, 3DCaptcha, ColorBlind, ChineseLigatures) designed to evaluate Multimodal Large Language Models (MLLMs) on synthetic images that are easy for humans but challenging for AI. The study reveals that state-of-the-art MLLMs exhibit catastrophic failures on these tasks, indicating a fundamental limitation in their vision tower’s generalization abilities rather than their reasoning capabilities. Fine-tuning the vision tower significantly improves performance, while in-context learning and language backbone fine-tuning do not, highlighting a critical gap in current MLLM visual perception.
Multimodal Large Language Models (MLLMs) have made incredible strides, showcasing powerful abilities in understanding and generating content across text and images. Much of the recent focus in AI research has been on enhancing their reasoning capabilities, allowing these models to tackle complex problems and answer intricate questions. However, a fundamental question has lingered: Can these advanced AI models truly perceive the world in the same intuitive way humans do?
A new research paper, titled “Pixels, Patterns, but No Poetry: To See The World like Humans,” shifts the spotlight from reasoning to a more foundational aspect: perception. Authored by Hongcheng Gao, Zihao Huang, Lin Xu, Jingyi Tang, and a team of researchers from various universities including the University of Chinese Academy of Sciences and Peking University, this preliminary study introduces a challenging new benchmark designed to test the visual perception limits of MLLMs. You can find the full paper here: Pixels, Patterns, but No Poetry: To See The World like Humans.
Instead of creating benchmarks that primarily evaluate reasoning, the researchers developed the Turing Eye Test (TET). This benchmark comprises four distinct diagnostic tasks, each designed with synthetic images that humans process effortlessly but pose significant challenges for AI. The tasks are:
HiddenText
This task involves images where text is subtly embedded within scenic backgrounds, appearing as shapes that resolve into readable text only when viewed holistically or at a specific scale. It tests the model’s ability for global pattern recognition in composite visuals.
3DCaptcha
Here, the challenge lies in recognizing characters that are curved and arranged in three-dimensional space, pushing the boundaries of spatial character recognition.
ColorBlind
Similar to traditional Ishihara tests, these charts are augmented with confounding colored dots that are chromatically similar to the central character, making it difficult for models to perceive the hidden pattern amidst visual noise.
Also Read:
- Bridging the Gap: How AI Models Can Learn from Their Own Mistakes in Image Generation
- Understanding Multimodal AI: Insights from the Visual Entailment Task
ChineseLigatures
This task features complex Chinese glyphs synthesized by combining and transforming multiple characters, requiring the model to decompose and recognize intricate character structures.
The findings from the Turing Eye Test are striking: state-of-the-art MLLMs, despite their impressive performance on other benchmarks, exhibit catastrophic failures on these perceptual tasks. Even increasing the number of attempts (Pass@K metrics) yields only marginal improvements, suggesting that the issue isn’t a lack of exploration in the reasoning space, but a deeper limitation in visual perception itself.
To understand why these models fail, the researchers conducted a preliminary analysis using Grad-CAM, a technique that visualizes which parts of an image the model focuses on. They found that models often fail to correctly locate the target regions in both their vision processing components (vision tower) and language processing components (language backbone). The vision encoder tends to focus on object-level features rather than the subtle textural features that form characters, while the language decoder often scatters its attention over irrelevant areas.
Further analysis through supervised fine-tuning (SFT) revealed a crucial insight: fine-tuning the vision encoder significantly improved performance on these tasks, enabling rapid adaptation. In contrast, fine-tuning only the language backbone or using in-context learning (providing examples within the prompt) showed little to no improvement. This strongly suggests that the bottleneck lies in the vision tower’s generalization abilities, rather than in the knowledge or reasoning capabilities of the language backbone.
The study also explored how image resolution affects performance on the HiddenText task. Downsampling images, which simplifies patch content and highlights character textures, improved performance. However, blurring, which introduces noise, led to inferior results. This aligns with how vision encoders process images in fixed-size patches, highlighting limitations in current visual encoding architectures.
In conclusion, the Turing Eye Test serves as a critical diagnostic tool, revealing a fundamental gap between current MLLMs and human-like visual perception. The research underscores the urgent need for improved visual generalization methods in MLLMs, perhaps by integrating reasoning capabilities directly into the perception stage. Future work will expand the TET with more diverse tasks and explore new methods to bridge this perception gap.
