TLDR: The MyCulture benchmark is introduced to evaluate Large Language Models (LLMs) on their understanding of Malaysia’s diverse culture, presented in Bahasa Melayu. It employs a novel open-ended multiple-choice question format, which significantly reduces guessing and reveals that LLMs’ cultural comprehension is often overestimated by traditional closed-form tests, showing a performance drop of at least 17%. The research highlights the need for culturally grounded and linguistically inclusive benchmarks to address biases in LLMs.
Large Language Models, or LLMs, have become incredibly powerful, capable of solving complex problems and generating human-like text. However, their training data, primarily sourced from the internet, is heavily dominated by high-resource languages like English and Chinese. This creates a significant challenge: LLMs often exhibit cultural biases, struggling to accurately represent and understand diverse cultural contexts, especially those tied to low-resource languages.
Introducing MyCulture: A New Benchmark for Malaysian Culture
To address this critical gap, researchers have introduced MyCulture, a groundbreaking benchmark designed to thoroughly evaluate LLMs on their understanding of Malaysian culture. Unlike many existing benchmarks that focus on more homogeneous cultural settings, MyCulture embraces the rich, multi-ethnic, and multi-religious fabric of Malaysia. It assesses LLMs across six key cultural pillars: arts, attire, customs, entertainment, food, and religion, with all questions presented in Bahasa Melayu, Malaysia’s national language.
Beyond Traditional Multiple-Choice: The Open-Ended Approach
One of MyCulture’s most innovative features is its novel open-ended multiple-choice question format. Traditional multiple-choice questions, while easy to create and score, can inadvertently inflate an LLM’s perceived performance. Models might guess correctly or rely on pattern recognition rather than true comprehension, simply because the answer options narrow down the possibilities. MyCulture tackles this by removing predefined options, forcing LLMs to generate responses independently. This significantly reduces the chance of random guessing and provides a more accurate measure of a model’s genuine cultural understanding.
The benchmark includes three types of open-ended questions: multi-answer, ordering, and matching. Each type is designed to increase the difficulty and demand a deeper understanding from the LLM. For instance, a multi-answer question might ask for several correct statements related to a topic, while an ordering question requires arranging events chronologically, and a matching question involves pairing items from two sets. This rigorous design ensures that only models with a true grasp of the cultural knowledge can succeed consistently.
Key Findings: The Illusion of Competence
The evaluation of 14 different LLMs, including popular models like GPT-4o and regional models like SEA-LION, revealed significant insights. While models generally performed better on the traditional closed-form multiple-choice questions, their accuracy dropped substantially—by at least 17%—when faced with the open-ended format. This stark difference highlights what the researchers call an “illusion of competence,” where LLMs appear to understand cultural knowledge well in simpler formats but struggle when required to generate answers without guidance. This underscores the urgent need for more robust and insightful evaluation methods like MyCulture.
Further analysis explored the impact of different factors on LLM performance:
-
Few-Shot Examples: Providing a few examples in the prompt did not significantly improve LLM performance on cultural understanding tasks, suggesting that cultural knowledge isn’t easily learned through limited examples.
-
Structured vs. Unstructured Outputs: Generally, LLMs performed better when guided to produce structured outputs (e.g., following a specific format), indicating that clear instructions can help models focus on generating the correct answer rather than lengthy, potentially inaccurate explanations.
-
Language Biases: Surprisingly, LLMs performed better when prompts were in high-resource languages like English and Chinese, even for questions about Malaysian culture, compared to prompts in Malay. This suggests that LLMs primarily acquire knowledge about Malaysian culture through content available in these dominant languages, highlighting the challenges faced by low-resource languages.
Also Read:
- Developing Curriculum-Aligned Math Assessments Using Generative AI in Malaysia
- Uncovering Hidden Geographic Biases in Large Language Models
The Path Forward for Culturally Aware AI
MyCulture represents a crucial step towards developing more culturally aware and inclusive LLMs. By providing a benchmark that accurately reflects the complexities of multi-ethnic societies and employs a more rigorous evaluation methodology, it pushes the boundaries of how we assess AI’s understanding of the world. This research emphasizes that for LLMs to truly serve diverse global communities, they must be developed and evaluated with culturally grounded and linguistically inclusive benchmarks. You can read more about this important work in the full research paper available here.
