TLDR: Ko-MuSR is a new benchmark for evaluating large language models’ multistep soft reasoning in long Korean stories, designed to avoid data contamination. It includes murder mysteries, object placements, and team allocations. Initial tests show multilingual models surprisingly outperform Korean-specialized ones, but carefully designed prompts significantly boost all models’ accuracy, sometimes to human levels. Smaller models, however, benefit less consistently from these prompts.
The development of artificial intelligence (AI) is increasingly moving beyond English-centric applications, highlighting a significant need for models that can effectively understand and process diverse languages. Addressing this, researchers have introduced Ko-MuSR, a novel benchmark specifically designed to evaluate the multistep soft reasoning capabilities of large language models (LLMs) within extensive Korean narratives. This benchmark is particularly noteworthy for its rigorous approach to minimizing data contamination, thereby ensuring a more accurate and reliable assessment of model performance.
Ko-MuSR draws inspiration from the existing MuSR benchmark but is meticulously crafted with fully Korean narratives, intricate reasoning chains, and multiple-choice questions. These components have undergone thorough verification by human annotators to guarantee both logical consistency and answerability. The primary goal of Ko-MuSR is to establish a robust framework for advancing Korean Natural Language Processing (NLP) by systematically evaluating how LLMs handle complex reasoning tasks in long Korean contexts and how various prompting strategies can enhance their effectiveness.
This new benchmark tackles several shortcomings identified in previous Korean evaluation datasets. Many existing Korean benchmarks fall short in assessing multistep reasoning outside of highly specialized fields like coding or mathematics. There’s also a noticeable gap in tasks that demand long-context reading, information extraction, and multi-hop reasoning in Korean. Furthermore, some current benchmarks are vulnerable to data contamination due to their reliance on publicly available materials frequently used in LLM training. Ko-MuSR circumvents these issues by creating logically sound passages and questions that are free from such contamination, with human annotators playing a crucial role in validating the derivability of answers from the provided texts.
The benchmark is structured around three distinct subtasks: murder mysteries, object placements, and team allocations. In the murder mysteries, models are tasked with identifying a killer from two suspects by analyzing evidence related to their means, motive, and opportunity. The object placements subtask challenges models to track where characters believe objects are located, taking into account who observed each movement. Lastly, the team allocations subtask requires models to assign three characters to a solo task and a collaborative task, optimizing the team composition based on individual skills and interpersonal compatibility.
Initial evaluations of Ko-MuSR involved four prominent LLMs: two multilingual models (Qwen3 32B and LLaMA 4 Scout Instruct) and two models specialized in Korean (Exaone 4.0 32B and HyperCLOVA X SEED Think 14B). A surprising outcome was that the multilingual models consistently outperformed their Korean-focused counterparts, even when performing reasoning tasks in Korean. This observation suggests a significant cross-lingual generalization of reasoning ability, particularly between English and Korean, implying that the underlying reasoning structure might be more critical than linguistic specialization for these complex narrative tasks.
Further research explored the impact of carefully designed prompting strategies, which included few-shot examples, Chain-of-Thought reasoning traces, and task-specific hints. These strategies led to a substantial increase in accuracy across all evaluated models, with some even achieving performance levels close to human capabilities. The hints, which provided detailed descriptions of how to approach each task, were found to contribute most significantly to these performance gains. This finding underscores that specialized models, even with comparatively smaller training budgets, can attain competitive reasoning performance in their target language through the application of structured prompting.
However, the study also highlighted an inconsistency in how smaller language models (SLMs) benefited from these detailed prompts. In certain instances, providing extensive prompts actually led to a decrease in accuracy for SLMs, indicating that prompting methods must be carefully adapted to the specific model size and task at hand. This aligns with previous research suggesting that the effectiveness of in-context examples tends to increase with model size.
Also Read:
- LongWeave: A New Standard for Assessing AI’s Long Text Capabilities
- Unpacking LLM Long-Context Abilities: Insights from the LooGLE v2 Benchmark
Ko-MuSR marks a vital advancement for Korean NLP, offering a systematic method to evaluate and enhance the long-context reasoning and prompting strategies of LLMs. The dataset, comprising 250 examples for each of the three subtasks, was generated through a controlled pipeline and meticulously refined via human review to ensure its factual and logical integrity. Researchers interested in delving deeper into this benchmark can access the code and data through the project’s GitHub repository. Explore the Ko-MuSR research paper for more details.
