TLDR: PakBBQ is a new, culturally adapted bias benchmark for Large Language Models (LLMs) in Pakistan. It features over 17,000 English and Urdu question-answer pairs across eight bias categories relevant to Pakistan’s unique social and linguistic context. Evaluations revealed significant performance gaps between English and Urdu, challenges with Pakistan-specific biases, and the effectiveness of explicit disambiguation and negative question framing in reducing stereotypical responses. The research highlights the necessity of culturally grounded bias evaluation beyond Western-centric frameworks.
Large Language Models (LLMs) are increasingly integrated into various applications, from conversational agents to medical assistance. However, a significant concern is their tendency to learn and perpetuate societal biases present in their training data. While many bias benchmarks exist, most are developed with Western, English-speaking contexts in mind, often overlooking the unique social, linguistic, and historical nuances of other regions.
Addressing Bias in Pakistani Contexts
Pakistan, with its rich cultural diversity, multi-ethnic population, distinct regional languages (Punjabi, Sindhi, Pashto, Balochi, alongside national Urdu), and varied religious landscape, presents a unique challenge for LLMs. Existing benchmarks like the Bias Benchmark for Question Answering (BBQ) or its Korean (KoBBQ) and Chinese (CBBQ) adaptations do not adequately capture biases relevant to Pakistan, such as those related to caste, sect, clan, or regional affiliations. Furthermore, the Urdu language itself encodes social hierarchies through pronouns, honorifics, and vocabulary choices, which can lead to structural biases if not properly addressed by LLMs.
Introducing PakBBQ: A Tailored Benchmark
To bridge this critical gap, researchers have introduced PakBBQ, the Pakistani Bias Benchmark for Question Answering. This new dataset is a culturally and regionally adapted extension of the original BBQ dataset, specifically tailored to the Pakistani context. PakBBQ comprises over 214 templates and 17,180 question-answer pairs in both English and Urdu. It covers eight bias dimensions relevant to Pakistan: age, disability, appearance, gender, socio-economic status, religious affiliation, regional affiliation, and language formality.
How PakBBQ Was Developed
The creation of PakBBQ involved a meticulous adaptation strategy. Templates from the original BBQ dataset were categorized into: Directly Translated (DT) for universally applicable scenarios, Target Modified (TM) for items requiring contextual changes to reflect Pakistani norms, and Simply Removed (SR) for irrelevant templates. Crucially, a new category, Newly Added (NA), was introduced to capture biases unique to Pakistan, such as sectarian affiliations, regional identities (e.g., Sindhi, Baloch), and minority religious groups. These NA templates were constructed using diverse sources like Pakistani social media, news, and academic literature, and then rigorously annotated by native speakers to ensure cultural relevance and consistency.
For translation into Urdu, the researchers opted to translate the generated JSONL data rather than the templates directly, overcoming challenges posed by placeholders and linguistic differences between English and Urdu. After evaluating multiple translation models, Google Translate API was chosen for its accuracy and contextual soundness.
Key Findings from Evaluations
The study evaluated several multilingual LLMs, including DeepSeek-V3, GPT-4.1 variants, and Gemini 2.0 Flash models, under zero-shot settings. The findings revealed several important insights:
- There was a consistent and substantial performance gap between English and Urdu, with models generally performing 7 to 17 percentage points worse in Urdu. This highlights the challenges of applying LLMs to low-resource languages, as most models are predominantly trained on English data.
- Models showed significantly lower accuracy on the Newly Added (NA) templates, which capture Pakistan-specific biases. This suggests that current LLMs struggle with culturally specific contexts outside their primary training scope, emphasizing that bias evaluation requires substantive cultural adaptation, not just linguistic translation.
- Explicit disambiguation, where the correct answer is clearly indicated, led to an average accuracy gain of 12% across all models. This suggests that providing clear contextual cues can help models overcome reliance on learned probabilistic defaults and reduce bias.
- Models achieved higher accuracy on negatively framed questions compared to positively framed ones, indicating a tendency to avoid stereotypes when questions are posed negatively. This effect was more pronounced in Urdu, suggesting that negative framing could be a simple yet effective prompt engineering technique for bias mitigation.
- Urdu models, on average, demonstrated stronger ‘counter-bias’ tendencies, particularly in disambiguated contexts. While this indicates progress in addressing discriminatory patterns, it also raises questions about whether models are developing genuine cultural sensitivity or simply overcorrecting with bias mitigation strategies.
Also Read:
- AI’s Sacred Challenge: Evaluating Language Models for Islamic Jurisprudence
- Navigating the Nuances: How LLMs Grapple with Ambiguous Graph Database Queries
Implications for Global AI Systems
The introduction of PakBBQ and its findings underscore the critical need to move beyond English-centric evaluation frameworks for AI bias. The research provides a replicable methodology for developing culturally specific bias benchmarks, emphasizing that meaningful cross-cultural evaluation requires creating entirely new categories of bias assessment. While the dataset currently focuses on Urdu and English, future work aims to include other major regional languages of Pakistan to ensure broader coverage and intersectional analysis.
The study also highlights practical strategies for mitigating bias, such as explicit disambiguation and negative question framing, which can be immediately applied in prompt engineering for more equitable multilingual AI deployment. However, the researchers caution that PakBBQ should be used responsibly for auditing and mitigation, not for fine-tuning models without safeguards, to prevent the reinforcement or amplification of existing prejudices.
