TLDR: HATS introduces a new Hindi Analogy Test Set (405 multiple-choice questions from Indian government exams) to evaluate reasoning in multilingual Large Language Models (LLMs). The study benchmarks state-of-the-art LLMs using various prompting strategies, including a novel grounded Chain of Thought approach. Findings indicate that LLMs perform best with English prompts, regardless of strategy, and highlight the need for better Hindi language resources for LLM evaluation.
Large Language Models (LLMs) have shown impressive capabilities in understanding and generating human language, but their reasoning abilities, especially in languages other than English, remain an area of active research. A new study introduces HATS, the Hindi Analogy Test Set, designed to specifically evaluate how well these advanced AI models can reason in Hindi.
Analogies are a powerful tool for testing reasoning because they require models to infer hidden relationships between concepts. While English LLMs are extensively evaluated, there’s a significant gap in understanding their performance in Indic languages like Hindi. HATS aims to bridge this gap by providing a much-needed resource for researchers and developers.
What is HATS?
HATS is a comprehensive test set comprising 405 multiple-choice analogy questions. These questions were not simply translated from English; instead, they were carefully curated from actual national and state-level administrative examinations and their preparatory materials in India. This ensures that the analogies are culturally relevant and reflect the semantic relationships specific to the Hindi language, unlike previous datasets that often relied on translated English analogies focusing only on syntactic relations.
The analogies in HATS follow the classic A:B::C:D format, meaning ‘A is to B as C is to D’. For example, one analogy from the set is: भोपाल (Bhopal): मध्य प्रदेश (Madhya Pradesh) :: भुवनेश्वर (Bhubaneshwar): ? The correct answer is उड़ीसा (Odisha), as Bhubaneshwar is the capital of Odisha, just as Bhopal is the capital of Madhya Pradesh.
Benchmarking LLMs on Hindi Reasoning
The researchers benchmarked three state-of-the-art multilingual LLMs: Aya-expanse-8B, Llama-3.1-8B, and Gemma-2-9B. These models were chosen for their strong performance in multilingual tasks and their accessibility for academic research. The evaluation involved various prompting strategies to assess how different ways of instructing the models affected their performance.
Three main prompting configurations were tested: Hindi-only prompts, English-only prompts, and a mixed approach (English system prompt with Hindi user prompt). The study also explored different Chain of Thought (CoT) prompting techniques, including a novel ‘Grounded Chain of Thought’ approach. This grounded approach guides the model’s reasoning process by providing a fixed sequence of steps based on cognitive theories of analogical reasoning.
Also Read:
- Unpacking How Large Language Models Grasp Story Analogies
- Unmasking True Mathematical Reasoning in AI: Introducing VAR-MATH
Key Findings and Challenges
The results revealed several interesting insights. Firstly, all models consistently performed best when both the system and user prompts were in English, even when solving Hindi analogies. This suggests that while multilingual models can process Hindi, their reasoning capabilities are often better elicited through English instructions. Gemma-2-9B emerged as the top performer, achieving the highest accuracy of 79.75% with the Grounded Zero-Shot Chain of Thought prompting in the English-only setting.
The study also highlighted limitations in current LLMs. Models struggled to correctly transfer analogical relationships from the A:B pair to the C:D pair, indicating challenges in structured reasoning. A significant issue observed was mistranslation of critical terms, particularly when models attempted to translate Hindi analogies internally. For instance, the Hindi word ‘ईंट’ (brick) was sometimes confused with ‘Eat’ due to phonetic similarity, leading to incorrect interpretations.
In conclusion, HATS provides a crucial resource for evaluating LLM reasoning in Hindi, addressing a significant gap in multilingual AI assessment. The findings underscore the need for continued development of multilingual models that can better understand and reason in non-English languages natively. The test set is publicly available for research purposes, and more details can be found in the full research paper: HATS: Hindi Analogy Test Set for Evaluating Reasoning in Large Language Models.
