TLDR: mSCoRe is a new benchmark for evaluating Large Language Models (LLMs) on multilingual and skill-based commonsense reasoning. It features a novel taxonomy of reasoning skills, a scalable data synthesis pipeline, and a complexity scaling framework. Experiments show current LLMs struggle with higher complexity and cultural nuances, often over-relying on logical reasoning, highlighting the need for more balanced training to improve their commonsense abilities.
Large Language Models, or LLMs, have made incredible strides in tackling complex tasks like math and coding. However, their ability to understand and apply everyday knowledge, especially across different languages and cultures, has remained a significant challenge. This area, known as commonsense reasoning, is crucial for AI to truly interact with the world in a human-like way.
Existing benchmarks designed to test commonsense reasoning often fall short. Many focus on a single language, like English, or rely on simple translations that miss cultural nuances. They also lack the ability to systematically increase task difficulty as LLMs become more capable, and they don’t offer a detailed way to analyze how models actually arrive at their answers.
To bridge these gaps, researchers Nghia Trung Ngo, Franck Dernoncourt, and Thien Huu Nguyen have introduced a new benchmark called mSCoRe: a Multilingual and Scalable Benchmark for Skill-based Commonsense Reasoning. This innovative framework aims to provide a more comprehensive and insightful evaluation of LLMs’ commonsense abilities.
What Makes mSCoRe Unique?
mSCoRe stands out with three core components designed to push the boundaries of LLM evaluation:
A New Taxonomy of Reasoning Skills: Unlike previous methods, mSCoRe breaks down reasoning into “atomic reasoning steps,” each categorized by a specific skill. This includes logical reasoning (inductive, deductive, abductive), contextual reasoning (analogical, counterfactual, probabilistic, temporal, spatial), and social & ethical reasoning (social, moral). This fine-grained analysis offers a deeper look into how models think.
A Robust Data Synthesis Pipeline: Instead of generating data from scratch, mSCoRe takes existing human-annotated datasets and systematically increases their complexity. This four-step process involves filtering initial questions, generating detailed reasoning paths, scaling up difficulty by adding context and options, and finally, making the commonsense context implicit within the question itself.
A Dynamic Complexity Scaling Framework: The benchmark can progressively increase the difficulty of questions, ensuring it remains challenging even as LLMs improve. This is achieved by requiring more reasoning steps and introducing more plausible, but incorrect, answer options.
Two Sides of Commonsense
mSCoRe is divided into two main subsets to cover different aspects of commonsense:
mSCoRe-G (General Commonsense): This part evaluates understanding of physical causality, temporal relationships, and basic world dynamics across five languages: English, German, French, Japanese, and Chinese.
mSCoRe-S (Social Commonsense): This subset focuses on understanding social interactions and cultural norms, using diverse cultural situations sourced from platforms like TikTok and Reddit.
Key Findings from Experiments
The researchers tested mSCoRe on eight state-of-the-art LLMs, including models like GPT-4o, OpenAI o1, LLaMA, and DeepSeek-R1. The results consistently showed that as question complexity increased, model performance declined, indicating that mSCoRe remains a significant challenge for current AI.
Interestingly, while larger models generally performed better, the improvements showed diminishing returns, suggesting that simply scaling up model size isn’t enough for advanced commonsense reasoning. The study also found that models often over-rely on logical reasoning, even when social or contextual reasoning would be more appropriate for a given task. This highlights a need for more balanced training approaches that encourage a wider range of reasoning skills.
Furthermore, the fine-grained skill taxonomy proposed by mSCoRe proved beneficial, leading to better performance compared to less structured approaches like standard Chain-of-Thought prompting. Models also struggled to dynamically adjust the depth of their reasoning, often using a fixed number of steps regardless of the question’s complexity.
Also Read:
- Assessing Emotional Intelligence in Large Language Models: Introducing MME-Emotion
- Evaluating Continuous Learning in Multimodal AI: Introducing MLLM-CTBench
Looking Ahead
The mSCoRe benchmark provides a valuable tool for evaluating and understanding the limitations of current LLMs in multilingual and culturally nuanced commonsense reasoning. The insights gained from this research point towards future directions for improving AI, emphasizing the need for more robust training methodologies that enhance models’ ability to utilize diverse reasoning skills and adapt their reasoning depth to task demands. You can read the full research paper for more details at this link.
