TLDR: Pragyaan introduces high-quality, culturally grounded post-training datasets (Pragyaan-IT and Pragyaan-Align) for 10 Indian languages. It uses a human-in-the-loop pipeline combining translation and synthetic expansion with manual refinement to overcome limitations of existing English-centric datasets, ensuring LLMs can better understand and respond to diverse Indian cultural contexts and tasks.
Large Language Models (LLMs) have become incredibly powerful, but their effectiveness in diverse linguistic and cultural contexts, especially for Indian languages, has been a significant challenge. The core issue lies in the scarcity of high-quality, culturally relevant data for post-training processes like instruction-tuning and preference-based learning.
Existing open-source datasets are often heavily English-centric, leading to performance gaps and a lack of cultural grounding when applied to Indian languages. Direct translations of English datasets frequently result in biases, errors, and a loss of cultural nuance. For instance, a prompt asking for “comfort food for a rainy day” might yield Western options like tomato soup, while an Indian user would expect responses like masala chai with pakoras or khichdi. Similarly, workout routines might default to squats instead of culturally familiar practices like Surya Namaskar.
Synthetic data generation, while promising, also faces hurdles in multilingual settings due to linguistic inaccuracies, grammatical inconsistencies, and potential hallucinations, leading to lower quality data.
Introducing Pragyaan: A New Approach to Indian Language Datasets
To address these critical gaps, researchers from Krutrim AI have introduced “Pragyaan,” a novel human-in-the-loop (HITL) pipeline designed to create reliable and diverse post-training data for Indian languages. This innovative approach combines translations with synthetic expansion, followed by meticulous manual editing.
The pipeline has been used to curate two significant datasets:
- Pragyaan-IT (22.5K examples): An instruction-tuning dataset aimed at improving an LLM’s ability to follow diverse prompts.
- Pragyaan-Align (100K examples): A preference dataset for Reinforcement Learning (RL)-based alignment, focusing on user intent, safety, and cultural grounding.
These datasets cover 10 Indian languages—Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, and Telugu—across 13 broad and 56 sub-categories, leveraging 57 diverse source datasets.
How Pragyaan Ensures Quality and Cultural Relevance
The methodology employs two complementary strategies:
- Translation with Human Refinement: English prompt-response pairs are translated into Indic languages using LLMs, then extensively refined by human annotators. This process creates both “Indic Generic Prompts/Responses” (direct translations) and “Indic Context Prompts/Responses” (culturally adapted versions).
- Synthetic Expansion with Human Refinement: A seed set of English prompts is expanded synthetically using a modified Self-Instruct pipeline. The resulting synthetic English prompts and their LLM-generated responses are then translated into Indic languages and refined by human annotators for correctness, clarity, and cultural grounding.
A crucial aspect of Pragyaan is its emphasis on “Indian Cultural Context.” The datasets are categorized into three levels:
- IC-1: Generic prompts and responses without explicit Indian anchoring.
- IC-2: Generic prompts yielding Indic-grounded responses (e.g., suggesting Indian breakfast items).
- IC-3: Explicitly Indic prompts eliciting fully Indic-based responses.
This multi-layered approach ensures that LLMs trained on Pragyaan data can understand and generate responses that are not just linguistically accurate but also culturally appropriate and relevant to Indian users.
Diverse Task Categories and Settings
The datasets are structured to cover a wide array of tasks, including reasoning, inference, natural language understanding and generation, question answering, dialogue, information extraction, mathematics, coding, and instruction following. Beyond these, it delves into culturally specific domains like Indian states, religions, and geo-political questions. To promote responsible AI, safety and non-compliance tasks are also included.
Tasks are further defined by several settings:
- Complexity: Ranging from easy (direct queries) to hard (multi-part questions requiring nuanced reasoning).
- Multi-Turn Interactions: From single-turn to extended five-turn conversations, crucial for dialogue systems.
- Instruction Following: Simple to complex constraints on response format and content.
- Safety: Ensuring models respond responsibly to sensitive or harmful content.
- Thinking Trails: Capturing explicit reasoning, from direct answers to Chain-of-Thought (CoT) and Self-Thinking.
The Human Touch: Ensuring High Quality
The “human-in-the-loop” refinement is central to Pragyaan’s quality. Annotators act as curators, verifying and adapting prompt-response pairs to align with designated configurations. They perform linguistic quality checks, ensuring fluency, grammatical accuracy, and appropriate response length. More importantly, they exercise creative judgment to refine awkward phrasings, restructure unclear outputs, and enrich responses with culturally relevant details, elevating the overall communicative value.
Also Read:
- Boosting Luxembourgish Language Models with a New Cross-Lingual Dataset
- Enhancing Cross-Lingual Abilities in LLMs with Code-Switching
Promising Results for Multilingual LLMs
A pilot study using Direct Preference Optimization (DPO) on the Pragyaan-Align dataset demonstrated significant improvements. Models like Krutrim-2-12B Instruct and Llama-3-8B Instruct showed enhanced alignment, with Krutrim-2-12B winning or drawing in over 60% of cases against its pre-DPO version. This confirms the dataset’s potential to build more inclusive and effective multilingual LLMs for the Indian context.
The Pragyaan datasets represent a crucial step towards bridging the linguistic and cultural gaps in LLM development, offering a robust foundation for models that truly understand and interact with the diverse Indian user base. For more details, you can read the full research paper here.
