TLDR: A new large-scale dataset, Bongard-RWR+, has been introduced to test AI’s abstract visual reasoning abilities using real-world-like images. It expands on previous Bongard Problem datasets by generating 5,400 instances of abstract concepts. Evaluations show that while current Vision Language Models (VLMs) can handle high-level visual concepts, they struggle significantly with discerning fine-grained details and generating natural language descriptions of these complex visual rules.
Artificial intelligence (AI) has made incredible strides in understanding and generating visual content. However, a significant challenge remains: abstract visual reasoning (AVR). This involves identifying and understanding abstract patterns and relationships in images, much like humans do when solving puzzles or making analogies. A classic testbed for AVR is Bongard Problems (BPs), which present a unique challenge for AI models.
Bongard Problems, first proposed in 1970, consist of two sets of six images, each set adhering to a different, abstract rule. The task is to figure out the hidden concept that separates the two sides and describe it. For instance, one side might show objects with vertical lines, while the other shows objects with horizontal lines. These problems are difficult because they require models to learn from very few examples (few-shot learning), understand context, and integrate visual perception with language generation.
Early Bongard Problem datasets used simple, black-and-white drawings, which didn’t fully capture the complexity of real-world scenes. Later efforts introduced real-world images, but often the concepts were high-level and easier for AI to grasp. The Bongard-RWR dataset was a step forward, using real-world images to represent the more abstract concepts found in the original BPs. However, its manual creation limited it to just 60 problems, which wasn’t enough for robust testing of modern AI models.
Introducing Bongard-RWR+
To address this limitation, researchers have introduced Bongard-RWR+, a new and significantly larger dataset designed to push the boundaries of abstract visual reasoning in AI. This dataset comprises 5,400 instances, all representing original abstract Bongard Problem concepts but using real-world-like images. The key innovation is an automated pipeline that leverages advanced Vision Language Models (VLMs) for dataset creation.
The creation process for Bongard-RWR+ involves several steps. First, a VLM called Pixtral-12B is used to describe existing manually curated images, generating both positive descriptions (what the image is) and negative descriptions (what it isn’t, based on the opposite concept). Then, a Text-to-Text (T2T) model augments these positive descriptions into many diverse variations, ensuring the core concept remains intact. Finally, a Text-to-Image (T2I) model, Flux.1-dev, synthesizes new images from these augmented descriptions. Crucially, human experts then manually verify these generated images to ensure they accurately reflect the intended concepts.
Evaluating AI Capabilities
The researchers used Bongard-RWR+ to conduct extensive evaluations of state-of-the-art VLMs, including InternVL2.5 78B, Qwen2-VL 72B-Instruct, LLaVA-Next 110B, and MiniCPM-o 2.6 8B. They tested these models across various Bongard Problem formulations:
- Binary Classification: Models classify a single test image or a pair of images to the correct side (left or right) of the problem.
- Text-based Classification: Images are first converted into natural language descriptions, and then a text-based model performs the classification.
- Concept Selection: Models choose the correct underlying concept from a set of candidate concepts.
- Concept Generation: The most challenging task, where models must generate a free-form natural language description of the concept themselves.
Also Read:
- Unpacking Object Reasoning: ORBIT Benchmark Exposes VLM Limitations
- Assessing AI’s Ability to Mimic Low Vision Perception
Key Findings and Limitations
The evaluations revealed several important insights into the current capabilities of VLMs in abstract visual reasoning. While VLMs showed some ability to recognize high-level, coarse-grained visual concepts, they consistently struggled with discerning fine-grained concepts. For example, concepts relying on subtle visual cues or precise spatial relationships proved particularly difficult.
Interestingly, the study found that an intermediate step of converting images into captions before reasoning (text-based classification) often improved model performance. This suggests that explicitly describing visual information can help models better ground their predictions. Additionally, larger models generally performed better, indicating that performance in AVR tasks does scale with model size to some extent. The research also showed that using grayscale images did not significantly impact performance, aligning with the idea that the concepts are primarily structural rather than color-dependent.
A crucial finding was that the generated real-world-like images in Bongard-RWR+ were as effective as real-world images for evaluating visual reasoning. This validates the automated generation pipeline and opens doors for creating even larger and more diverse datasets in the future. However, the most challenging task, free-form concept generation, consistently yielded low scores across all models, highlighting a significant gap in their ability to articulate complex visual rules in natural language.
In conclusion, Bongard-RWR+ serves as a robust and challenging benchmark for abstract visual reasoning. The findings underscore that while current VLMs are powerful, they still have significant limitations in understanding and articulating fine-grained visual concepts, especially compared to human capabilities. This work, detailed further in the research paper available at this link, paves the way for future research to bridge this gap in AI’s visual intelligence.
