TLDR: A new benchmark, VLMCountBench, reveals that while Vision-Language Models (VLMs) can count single types of objects reliably, they significantly fail at compositional counting (counting multiple types of objects simultaneously). The study, which evaluated state-of-the-art VLMs, found that performance degrades with increased object types and that prompt refinements or visual perturbations like color and size changes often do not improve, and sometimes even worsen, counting accuracy. This highlights a fundamental limitation in current VLMs’ quantitative understanding.
Vision-Language Models (VLMs) have become a cornerstone of modern artificial intelligence, showcasing remarkable capabilities in understanding and processing both visual and linguistic information. These models, trained on vast amounts of web data, excel in tasks like image and video understanding, complex visual reasoning, and even embodied AI. However, a recent study delves into a fundamental question: how well can these advanced models actually count objects?
A new benchmark called VLMCountBench, introduced in a paper titled “Your Vision-Language Model Can’t Even Count to 20: Exposing the Failures of VLMs in Compositional Counting”, aims to answer this question. Designed under a minimalist setting, this benchmark uses basic geometric shapes like triangles, circles, and squares, and their combinations, to focus purely on counting tasks without interference from other complex factors. The researchers, Xuyang Guo, Zekai Huang, Zhenmei Shi, Zhao Song, and Jiahao Zhang, systematically studied how simple properties such as color, size, and even prompt wording affect VLM performance.
The findings are quite revealing. While VLMs can count reliably when only one type of shape is present in an image, they struggle significantly when multiple shape types are combined – a task the researchers call “compositional counting.” This limitation persists even with small numbers of objects and minimal visual complexity. For instance, models that performed well when counting only circles saw a substantial drop in accuracy when asked to count circles, squares, and triangles simultaneously.
The study evaluated several state-of-the-art VLMs, including both open-source models like Gemma 3 27B and Qwen 2.5 72B, and closed-source models such as Gemini 2.5 Flash and GPT-4o. Even the best-performing models achieved only modest accuracy in compositional counting. Qwen 2.5 72B, for example, had an accuracy of 0.60 for single-shape counting but dropped to 0.45 for three-shape counting. This indicates a significant gap between current VLM capabilities and the reliable counting ability needed for practical applications.
Interestingly, the research also explored the impact of visual variations. Randomizing colors and sizes of the shapes had mixed effects. Some models, like GLM4.5v and GPT-4o, surprisingly showed improved accuracy with color variations, possibly because distinct colors made objects easier to differentiate. However, others, such as Ernie 4.5 and Qwen2.5 72B, saw their performance decline, suggesting they might rely on specific color patterns learned during training. Size variations generally had a smaller impact on performance across models.
The researchers also investigated whether refining prompts could help. They tested “spatial decomposition” (counting objects in halves of the image and then combining) and “type decomposition” (counting one type of object first, then another). Counter-intuitively, these prompt refinements generally led to a slight reduction in accuracy and an increase in relative error compared to the original, simpler prompts. This suggests that breaking down the task into more steps might introduce additional errors or complicate the VLM’s inference process, rather than simplifying it.
Also Read:
- The Evolving Landscape of AI Evaluation: From Simple Recognition to Complex Reasoning
- Evaluating AI’s Understanding of Physical Privacy: A New Benchmark Reveals Critical Gaps
In conclusion, VLMCountBench highlights a fundamental empirical limitation of current vision-language models in compositional counting. Despite their impressive overall capabilities, accurately quantifying multiple types of objects in a single visual scene remains a significant challenge. This research underscores the need for future developments to enhance the robust counting abilities of VLMs, moving beyond simple recognition to more precise quantitative understanding.
