TLDR: Researchers from the Indian Institute of Technology Patna have introduced ‘Drishtikon,’ a pioneering multimodal and multilingual benchmark. Comprising over 64,000 text-image pairs across 15 Indian languages and all states/union territories, it aims to rigorously evaluate the cultural understanding of generative AI systems, revealing current limitations, especially in low-resource languages and less-documented traditions.
A groundbreaking new benchmark, ‘Drishtikon,’ has been developed to address a critical gap in artificial intelligence: the ability of language models to comprehend and reason about diverse cultures. Unveiled by Arijit Maji, Raghvendra Kumar, and Akash Ghosh, all from the Indian Institute of Technology Patna, alongside Anushka, Nemil Shah, Abhilekh Borah, Vanshika Shah, and Nishant Mishra, and Sriparna Saha, this benchmark is the first of its kind to focus exclusively on Indian culture.
The ‘Drishtikon’ dataset is a robust collection of over 64,000 aligned text-image pairs, offering deep and fine-grained coverage across India’s vast and diverse regions. It spans 15 languages, encompassing all states and union territories, and captures a rich array of cultural themes. These themes include festivals, traditional attire, diverse cuisines, various art forms, and historical heritage, among many others.
The primary objective of ‘Drishtikon’ is to provide a comprehensive tool for evaluating the cultural understanding of generative AI systems. Existing benchmarks often possess a generic or global scope, lacking the specific cultural depth required to adequately assess AI’s capabilities in this nuanced area.
In their evaluation, the researchers tested a wide range of vision-language models (VLMs), including open-source models (both small and large), proprietary systems, reasoning-specialized VLMs, and models specifically focused on Indic languages. These evaluations were conducted across both zero-shot and chain-of-thought settings.
The results of these evaluations have exposed significant limitations in current models’ ability to reason over culturally grounded, multimodal inputs. This deficiency was particularly evident for low-resource languages and less-documented traditions within India. For instance, while proprietary models like GPT-4o mini demonstrated strong performance, likely benefiting from extensive instruction tuning and alignment, compact instruction-tuned models such as SmolVLM-256M-Instruct and InternVL3-1B also delivered competitive results, underscoring the potential of efficiency-aware architectures for culturally rich tasks.
Also Read:
- IIIT Hyderabad Spearheads India’s Generative AI Ambitions with Key Contributions to BharatGen Initiative
- Unmasking Cultural Positioning Bias in AI: How LLMs Adopt a Mainstream Lens
‘Drishtikon’ is poised to fill a vital void in inclusive AI research, offering a much-needed testbed to foster the development of culturally aware and multimodally competent language technologies. Its introduction marks a significant step towards building AI systems that can better understand and interact with the world’s diverse human cultures.
