TLDR: IndicVisionBench is a new large-scale benchmark for Vision-Language Models (VLMs) focused on the Indian subcontinent. It evaluates VLMs on tasks like Visual Question Answering (VQA), Optical Character Recognition (OCR), and Multimodal Machine Translation (MMT) across 10 Indian languages and English. Utilizing 5,000 images and over 37,000 question-answer pairs covering 13 culturally grounded topics, the research reveals substantial performance gaps in current VLMs for culturally diverse and low-resource language settings. The benchmark highlights the critical need for more inclusive AI development to better understand global cultural and linguistic nuances.
Vision-language models (VLMs) have shown impressive capabilities across various multimodal tasks, but most existing evaluation benchmarks primarily focus on Western cultures. This leaves a significant gap in understanding how these models perform in the rich and diverse cultural and linguistic landscapes found globally, particularly in a country like India.
To address this crucial gap, researchers have introduced IndicVisionBench, the first large-scale benchmark specifically designed for the Indian subcontinent. This new benchmark aims to systematically evaluate VLMs on their cultural and multilingual understanding, providing a much-needed resource for more inclusive AI development.
A Comprehensive Look at Indian Diversity
IndicVisionBench is a robust evaluation suite that covers English and 10 prominent Indian languages, including Hindi, Bengali, Tamil, Malayalam, Telugu, Marathi, Kannada, Gujarati, Punjabi, and Oriya. It spans three key multimodal tasks:
- Optical Character Recognition (OCR): Assessing the models’ ability to accurately read and interpret text from images in various Indic scripts.
- Multimodal Machine Translation (MMT): Evaluating how well models can translate captions from English into Indic languages while understanding the visual context.
- Visual Question Answering (VQA): Testing the models’ comprehension of images by answering questions across six different types, ranging from simple short answers to complex adversarial questions that challenge cultural knowledge.
The benchmark comprises a substantial collection of 5,000 unique images and over 37,000 question-answer pairs. These are carefully curated across 13 culturally grounded topics, such as Food, Lifestyle, Literature, Music and Dance, Religion, Customs, Festivals, Heritage, Economy, Media, Architecture, Sports, and Notable Figures. This extensive coverage ensures that the evaluation delves deep into India’s diverse traditions and contexts.
A unique aspect of IndicVisionBench is the release of a paired parallel corpus of annotations across 10 Indic languages. This resource is invaluable for researchers looking to analyze cultural and linguistic biases present in current VLMs.
How the Benchmark Was Created
The creation of IndicVisionBench involved a meticulous process to ensure reliability and cultural fidelity. Images were collected through a combination of controlled crowd-sourcing and large-scale web crawling, focusing on Creative Commons–licensed content. Human verification and correction were applied at every stage of data collection.
For VQA, initial question-answer pairs were generated using advanced language models like Gemini-1.5-Flash and Gemini-2.5-Flash, which were then refined by human reviewers for factual accuracy and cultural alignment. This process included creating challenging ‘adversarial questions’ that embed false assumptions, pushing models to demonstrate deeper cultural understanding beyond surface-level recognition.
The MMT track involved manually annotating English captions translated into 10 Indic languages, ensuring that the translations preserved meaning and cultural nuances within the image context. For OCR, a multilingual corpus was built from Wikisource, pairing high-resolution scans with human-reviewed text to create a high-fidelity ground truth.
Key Findings: Performance Gaps and Biases
The research evaluated a broad spectrum of 8 state-of-the-art VLMs, including proprietary closed-source systems (like Gemini-2.5 Flash and GPT-4o) and open-weight models (such as Gemma-3-27B and LLaMA-4-Maverick-17B). The results revealed significant performance gaps, underscoring the limitations of current VLMs in culturally diverse contexts.
Proprietary models, particularly Gemini-2.5, consistently demonstrated superior performance across all tasks and question types. However, even these advanced models struggled with adversarial questions, highlighting the inherent difficulty in probing nuanced cultural knowledge.
The study also observed considerable performance drops for low-resource languages and culturally specific content. Furthermore, experiments confirmed that visual input is crucial for VQA tasks, as removing images led to a substantial decrease in accuracy. The research also identified regional-language biases, with Odia content proving consistently challenging for all models.
Also Read:
- Evaluating AI’s Coding Prowess in Ukrainian: Introducing UA-Code-Bench
- DeepEyesV2: Enabling AI to Actively Use Tools for Complex Multimodal Tasks
Paving the Way for Inclusive AI
The introduction of IndicVisionBench marks a significant step towards building more inclusive and globally robust multimodal systems. By centering cultural diversity and multilinguality, this benchmark establishes a reproducible evaluation framework that will help researchers and developers identify and address the limitations of current VLMs. This work is crucial for fostering AI technologies that can truly understand and interact with the world’s rich tapestry of cultures and languages.
For more details, you can refer to the full research paper: INDICVISIONBENCH: BENCHMARKINGCULTURAL ANDMULTILINGUALUNDERSTANDING INVLMS.
