TLDR: A study evaluated Diffusion Classifier, CLIP, and ViLT on their ability to understand how concepts combine (compositionality) in images. While Diffusion Classifier and ViLT showed good performance on binding attributes to single or two objects, all models struggled significantly with understanding spatial relationships like “left” and “right,” especially when distinguishing between similar but distinct scenarios. This suggests current models often rely on recognizing individual objects rather than truly grasping how they relate to each other, highlighting a need for better pre-training to develop these fundamental semantic abilities.
Vision-language models (VLMs) have made impressive strides in recent years, but a new research paper titled “Evaluating Compositional Generalisation in VLMs and Diffusion Models” by Beth Pearson, Bilal Boulbarss, Michael Wray, and Martha Lewis, delves into a critical area where these advanced AI systems still fall short: compositional understanding. This refers to the human-like ability to combine familiar concepts in new ways to interpret novel situations, a skill crucial for true intelligence.
The core problem is that VLMs often struggle to correctly interpret images where attributes and objects are combined in new ways. For instance, if a VLM sees an image of a red cube and a blue cylinder, it might incorrectly label it as a red cylinder or a blue cube. This suggests that the model is treating the image like a “bag-of-words,” recognizing individual elements but failing to bind them correctly – a significant limitation in grasping compositional semantics.
Models Under the Microscope
The researchers put three prominent models to the test: CLIP, ViLT, and the Diffusion Classifier. CLIP (Contrastive Language–Image Pre-training) and ViLT (Vision-and-Language Transformer) are well-known discriminative vision-language models. The Diffusion Classifier, on the other hand, is a newer approach built from generative diffusion models, which have recently gained attention for their impressive image generation capabilities and their potential as zero-shot classifiers.
The Experimental Setup: An Extended Benchmark
To rigorously evaluate compositional generalization, the team extended the Concept Binding Benchmark. This benchmark uses simple, synthetic images of 3D shapes (cubes, cylinders, spheres, and cones) in various colors. This controlled environment helps to eliminate spurious correlations often found in complex real-world images, allowing for a direct test of compositional understanding. The benchmark is divided into three main tasks:
- Single-Object Task: Tests the model’s ability to recognize attribute-object pairs (e.g., “red cube”).
- Two-Object Task: Involves images with two distinct objects, differing in both shape and color (e.g., “a blue cube and a red sphere”).
- Relational Task: Focuses on understanding spatial relationships between two distinct objects, specifically “left” and “right” (e.g., “cube left sphere”).
The experiments were conducted in two settings: Zero-Shot Learning (ZSL) and Generalised Zero-Shot Learning (GZSL). ZSL evaluates a model’s ability to recognize entirely unseen classes, while GZSL is more challenging, requiring models to discriminate between both seen and unseen classes during testing. The GZSL setting is particularly important for real-world applications, as models often need to generalize to new concepts while still recognizing familiar ones.
Key Findings: A Mixed Bag of Performance
The results revealed a nuanced picture of the models’ compositional abilities:
- Single-Object Task: After fine-tuning, the Diffusion Classifier showed the best overall accuracy, demonstrating a strong ability to generalize to unseen color-shape combinations. CLIP also performed well, but ViLT struggled with out-of-distribution (OOD) examples, failing to generalize from familiar components.
- Two-Object Tasks (ZSL & GZSL): ViLT, surprisingly, emerged as the strongest performer in these tasks, achieving over 99% accuracy in ZSL and maintaining high performance in GZSL. This suggests ViLT is less biased towards seen labels and has strong attribute-object binding capabilities. However, both fine-tuned CLIP and Diffusion Classifier showed a decrease in performance on OOD test sets, indicating limitations in robust compositional understanding for more complex scenarios.
- Relational Tasks (ZSL & GZSL): This is where all models hit a significant roadblock. They performed poorly across the board, especially in the GZSL setting with “hard negatives” (e.g., distinguishing “cube left sphere” from “cube right sphere”). This substantial drop in performance compared to the two-object tasks suggests that systematically combining objects with relations is much harder than combining color-object pairs. The models appear to rely more on recognizing individual objects rather than truly understanding the spatial relationships between them.
Why the Struggle with Relations?
An analysis of CLIP’s embeddings provided further insight. The text embeddings for relational concepts like “left” and “right” were found to be overly similar and not sufficiently disentangled. This means the model struggles to differentiate between prompts that describe different arrangements of objects, directly contributing to its inability to correctly interpret images with varying spatial relations.
Also Read:
- Uncovering a Factual Recall Gap in Vision Language Models
- Bridging Vision and Language: The Evolution of Object Detection with Advanced AI Models
Implications for Future AI
The research highlights that current VLMs may be relying on “shortcuts,” such as simple object recognition, rather than developing structured, compositional representations of visual scenes. While fine-tuning strategies can offer some improvements (the authors found their positive-only fine-tuning for CLIP led to better generalization), the fundamental limitation in relational reasoning persists.
The authors argue that these fundamental semantic abilities, like compositional understanding, should be addressed at the pre-training stage of these models, rather than expecting users to fine-tune for them. Future work could involve expanding the benchmark to include more attributes (like material or size) or more than two objects, further pushing the boundaries of compositional reasoning in AI. For more details, you can read the full research paper here.
