TLDR: A new research paper introduces CultSportQA, a large multilingual and multicultural benchmark designed to assess language models’ understanding of traditional sports. Covering 60 countries and 11 languages with 33,000 text and image-based questions, it addresses the gap left by existing benchmarks that primarily focus on globally popular sports. Evaluations show that while larger language models perform better, there are still significant challenges in culturally nuanced sports reasoning, especially across diverse languages and question types. The dataset aims to foster more inclusive and culturally aware AI development.
Language models, while advanced, often fall short when it comes to understanding the rich tapestry of traditional and indigenous sports from around the world. Their training data typically focuses on globally popular sports, leading to a significant gap in their ability to recognize and reason about diverse cultural contexts in athletics.
To address this crucial oversight, a new benchmark called CultSportQA has been introduced. This comprehensive dataset is designed to evaluate how well language models understand traditional sports across a vast array of cultures. It covers 60 countries and six continents, encompassing four distinct cultural categories of sports.
The CultSportQA dataset is substantial, featuring approximately 33,000 multiple-choice questions. These questions are presented in both text and image formats, making it a multimodal benchmark. Each question is further categorized into three key types: history-based, rule-based, and scenario-based, challenging AI models to engage with sports knowledge at different levels of complexity.
Researchers evaluated a diverse range of models on CultSportQA, including Large Language Models (LLMs), Small Language Models (SLMs), and Multimodal Large Language Models (MLLMs). They employed various prompting techniques, such as zero-shot, few-shot, and Chain-of-Thought (CoT) learning, to thoroughly assess model performance.
The motivation behind CultSportQA stems from the limitations of existing benchmarks. Previous datasets like SportQA and SportU primarily focused on well-known sports and were often limited to English. CultSportQA fills this void by providing the largest multicultural and multilingual sports benchmark to date, capturing the nuanced cultural aspects of sports reasoning across multiple languages and visual contexts.
How the Dataset Was Built
The creation of CultSportQA involved a meticulous, multi-phase process to ensure high quality and comprehensive coverage. Domain experts and native-speaking annotators from 11 countries across three continents played a vital role. These annotators, fluent in their local languages and aware of cultural nuances, contributed to data collection, question formulation, and manual translation.
Data was gathered from six credible sources, including Wikipedia, National Heritage and Sports Boards, Local Sports Blogs, Cultural Journals, News Outlets, and Academic Publications. This diverse sourcing ensured a rich collection of information on traditional sports from India, Pakistan, Bangladesh, Italy, France, China, Thailand, Indonesia, Sudan, Ethiopia, and Germany.
The annotation process was structured with a team of experts, where two-thirds created questions and the remaining annotators validated and filtered them. Questions had to be complete, self-contained, and culturally aligned with the sport and country. They followed a multiple-choice format with four options, only one of which was correct. A rigorous quality assurance process, including cross-validation and bias mitigation, was implemented to maintain accuracy and cultural sensitivity.
Key Findings from the Evaluation
The evaluation revealed several important trends. Overall, Large Language Models consistently outperformed other model types. GPT-4o and Llama-3.1-70B demonstrated superior accuracy across all evaluation categories, excelling in language-based tasks and different question types. GPT-3.5 also performed strongly, particularly in history-based and rule-based questions.
Among Small Language Models, Mistral-7B achieved the highest performance, showcasing strong generalization capabilities, especially with few-shot learning. Gemma-7B and Phi-3-medium followed closely. For Multimodal Large Language Models, InstructBLIP emerged as the top performer, demonstrating robust reasoning and adaptability in image-based questions.
Performance varied across the 11 languages included in the benchmark. GPT-4o led in most languages, while models like BART struggled, particularly with non-Latin scripts such as Amharic and Thai. The study also highlighted that few-shot learning and Chain-of-Thought prompting generally improved model performance across all categories.
Also Read:
- Assessing Cultural Understanding in Multimodal AI with a New Comic-Based Benchmark
- Beyond English: Understanding AI’s Reasoning Across Multiple Languages
Impact and Future Directions
CultSportQA establishes a new standard for assessing AI’s ability to understand and reason about traditional sports in a more inclusive and culturally aware manner. By highlighting the limitations of current models in culturally nuanced domains, it encourages the development of more equitable AI applications. The dataset is publicly available for further research and development. [cite: RESEARCH_PAPER_URL]
While significant, the study acknowledges limitations, including a focused geographic scope, a finite number of traditional sports covered (84), and a limited set of languages and modalities. Future work aims to expand the dataset to include more countries, languages, and diverse question-answering tasks to further enhance its impact on inclusive AI research.
