TLDR: A new benchmark called RotBench reveals that while Multimodal Large Language Models (MLLMs) can identify right-side-up and often upside-down images, they consistently struggle to distinguish between 90° and 270° rotations. Neither auxiliary information nor chain-of-thought prompting significantly resolves this spatial reasoning gap, highlighting a fundamental limitation compared to human perception.
Multimodal Large Language Models (MLLMs) have made incredible strides in understanding and generating content across various data types, especially images and text. They excel at complex visual tasks like image-text retrieval and visual question answering. However, a recent study introduces a new benchmark, RotBench, to explore a seemingly simple yet fundamental challenge for these advanced AI models: accurately identifying image rotation.
The research, detailed in the paper RotBench: Evaluating Multimodal Large Language Models on Identifying Image Rotation, investigates how well MLLMs can determine if an image has been rotated by 0°, 90°, 180°, or 270° counter-clockwise. This task demands robust visual reasoning to detect rotational cues and understand spatial relationships within images, regardless of their orientation. Humans can effortlessly recognize if an image is upside down or sideways, but can AI do the same?
Introducing RotBench
To evaluate this capability, the researchers developed RotBench, a benchmark comprising 350 carefully selected images. These images, which include lifestyle, portrait, and landscape scenes, underwent a rigorous two-stage manual filtering process. This ensured that each image, when rotated, presented clear and distinguishable orientations, preventing ambiguity that might confuse human evaluators, let alone AI.
The Experiment and Surprising Results
The study tested several state-of-the-art MLLMs, including prominent models like GPT-5, o3, and Gemini-2.5-Pro, alongside open-source alternatives like Qwen-2.5-VL-7B-Instruct. Each image from RotBench was presented to the models in its four rotated states (0°, 90°, 180°, and 270°), framed as a multiple-choice question. The researchers also explored whether providing auxiliary information (such as captions, bounding boxes, scene graphs, depth maps, or segmentation maps) or using Chain-of-Thought prompting could improve performance.
The findings revealed a significant gap in MLLMs’ spatial reasoning:
-
Right-Side-Up Images (0°): All models performed exceptionally well, consistently identifying unrotated images with near-perfect accuracy. This is likely because models are predominantly trained on upright images.
-
Upside-Down Images (180°): Proprietary models generally showed robust performance, often achieving accuracies well above chance, indicating a reliable ability to recognize upside-down images.
-
Sideways Images (90° and 270°): This is where MLLMs struggled most. All models exhibited substantial difficulty distinguishing between 90° and 270° rotations. Confusion matrix analysis showed frequent misclassifications between these two orientations, suggesting a fundamental challenge in differentiating clockwise from counter-clockwise rotations.
Limited Impact of Auxiliary Information and Prompting
Surprisingly, providing additional auxiliary information to the models did not consistently or meaningfully improve their performance. In some cases, it even led to a slight degradation. Chain-of-Thought prompting, which encourages models to show their reasoning steps, yielded mixed results; while it sometimes improved accuracy for 180° rotations, its effect on 90° and 270° rotations was inconsistent, often improving one at the expense of the other.
An interesting approach involved presenting the models with a “rotation grid” – the input image along with its 90°, 180°, and 270° rotations simultaneously. This method helped stronger “reasoning” models like o3 and Gemini-2.5-Pro improve their performance on 90° and 270° rotations. A further refinement using a majority voting system across these rotations also showed gains for weaker models, but this approach requires multiple model calls and assumes prior knowledge of all possible orientations.
Fine-Tuning Challenges
The researchers also conducted fine-tuning experiments to see if specialized training could mitigate these issues. While fine-tuning significantly improved the identification of 180° images, it did not resolve the challenge of distinguishing between 90° and 270° rotations. The accuracy for these two orientations showed an oscillating pattern during training, suggesting the models struggled to find a stable solution, possibly due to inherent representational limitations in their visual encoders.
Also Read:
- Unpacking Object Reasoning: ORBIT Benchmark Exposes VLM Limitations
- Evaluating AI Hallucinations in First-Person Video Analysis
Conclusion
The RotBench study highlights a significant gap between the spatial reasoning capabilities of current MLLMs and human perception. Despite their advancements in other complex visual tasks, these models consistently underperform when it comes to accurately identifying image orientation, particularly distinguishing between 90° and 270° rotations. These findings underscore the critical need for future research and development to integrate better rotation-awareness into modern MLLM training pipelines, moving towards AI that can perceive the world with a more human-like understanding of spatial relationships.
