TLDR: ORBIT is a new benchmark designed to systematically evaluate how Vision-Language Models (VLMs) reason about object properties in images. It features 360 images across three types (photographic, animated, AI-generated), three reasoning levels (recognition, inference, counterfactual), and four object property dimensions (physical, taxonomic, functional, relational). Experiments with 12 state-of-the-art VLMs show significant limitations compared to humans, particularly with realistic images, counterfactual reasoning, and higher counts, indicating a need for VLMs to develop stronger object abstraction and reasoning capabilities.
Vision-Language Models (VLMs) have shown impressive capabilities in tasks like visual question answering (VQA). However, a new research paper introduces a benchmark called ORBIT, which suggests that these models still struggle with abstracting and reasoning over depicted objects in a way humans do.
The paper, titled “ORBIT: An Object Property Reasoning Benchmark for Visual Inference Tasks,” by Abhishek Kolari, Mohammadhossein Khojasteh, Yifan Jiang, Floris den Hengst, and Filip Ilievski, highlights a critical gap in current VLM evaluation. Existing VQA benchmarks often blend perception and reasoning, focus on a limited set of object attributes, and lack diversity in reasoning and image categories. ORBIT aims to provide a more systematic and comprehensive evaluation.
ORBIT’s Comprehensive Framework
ORBIT introduces a structured evaluation framework built around three core components:
First, it uses three representative image types: Photographic (realistic, complex scenes), Animated (simplified, stylized), and AI-generated (testing robustness to domain shifts and implausible objects). This diversity helps assess how well VLMs generalize across different visual domains.
Second, the benchmark defines three levels of reasoning complexity: Direct Recognition, which involves identifying basic visual elements or taxonomic categories (e.g., counting mammals); Property Inference, requiring deeper abstraction for functional or relational properties (e.g., identifying means of transportation); and Counterfactual Reasoning, the most challenging, which involves reasoning about hypothetical changes in the image (e.g., if half the clocks were removed, how many circular objects would remain?).
Third, ORBIT focuses on four object property dimensions, drawing from prior work on commonsense reasoning: Physical (e.g., shape, material, part-whole relationships), Taxonomic (semantic categories like ‘mammals’ or ‘furniture’), Functional (capabilities or utilities, like ‘means of transportation’), and Relational (how objects interact or are grouped, such as ‘on top of’ or ‘couples’).
Benchmark Construction and Key Findings
The ORBIT benchmark comprises 360 images paired with a total of 1,080 count-based questions. The dataset was created using a semi-automatic procedure, where initial questions were generated by large multimodal language models (MLLMs) and then extensively refined and quality-assured by human annotators to ensure precision and reduce ambiguity.
Experiments were conducted with 12 state-of-the-art VLMs in zero-shot settings. The results revealed significant limitations compared to human performance. While humans achieved an average accuracy of 74%, the best-performing VLM only reached 40% accuracy. This highlights a substantial gap in current AI capabilities.
Specifically, VLMs struggled most with:
- Realistic (photographic) images: They performed consistently worse on complex, noisy real-world scenes compared to cleaner animated or AI-generated images.
- Counterfactual reasoning: Questions involving hypothetical changes or out-of-context scenarios proved particularly difficult.
- Physical and functional properties: Models had a weaker grasp on these dimensions compared to taxonomic and relational questions.
- Higher counts: VLMs showed a bias towards low, more frequent counts, with accuracy dropping significantly for counts over 5, unlike humans who showed uniform accuracy across counts.
- Undercounting: A general bias towards underestimation was observed across many models.
Even when allowing for a small counting error (off-by-1 accuracy), the best model only reached 73%, indicating that while predictions might be close, precise numerical grounding remains a challenge.
Also Read:
- Assessing AI’s Ability to Mimic Low Vision Perception
- PMTFR: A Novel Framework for Enhanced Composed Image Retrieval
Implications and Future Directions
The ORBIT benchmark points to a clear need for developing new methods that enhance VLMs’ ability to perform scalable benchmarking, generalize annotation guidelines, and explore additional reasoning capabilities. The researchers note that current generative AI models still lack the accuracy and diversity for autonomous question generation, requiring substantial human effort for scaling up such benchmarks.
The ORBIT benchmark and experimental code are made available to support future research in this critical area of VLM development. For more details, you can refer to the full research paper: ORBIT: An Object Property Reasoning Benchmark for Visual Inference Tasks.
