TLDR: MultiZebraLogic is a new multilingual benchmark for evaluating large language models (LLMs) on logical reasoning. It uses ‘zebra puzzles’ in nine Germanic languages, varying difficulty through puzzle size and ‘red herrings’ (uninformative clues). The study found that red herrings significantly increase difficulty for reasoning models like o3-mini, while language and cultural themes had less impact, suggesting generalizable reasoning abilities. Datasets and generation code are publicly available.
A new research paper introduces MultiZebraLogic, a groundbreaking benchmark designed to thoroughly evaluate the logical reasoning capabilities of large language models (LLMs) across a diverse range of languages. This initiative addresses a critical gap in current LLM evaluation, where existing benchmarks either cover many languages but lack logical reasoning tasks, or focus exclusively on English for such evaluations.
The core of MultiZebraLogic revolves around ‘zebra puzzles,’ a specific type of constraint satisfaction problem. These puzzles are ideal for benchmarking because they are relatively simple to generate but demand multiple steps of logical deduction to solve, making them an excellent test for LLMs. The researchers explored several methods to adjust the difficulty of these puzzles, including altering their size, incorporating ‘red herrings’ (clues that are uninformative or misleading), and varying the types of clues and cultural themes used.
Currently, the benchmark supports nine Germanic languages: English, Danish, Swedish, Norwegian Bokmål, Norwegian Nynorsk, Faroese, Icelandic, German, and Dutch. A significant contribution of this work is the release of the source code for puzzle generation, which is designed for easy adaptation to include even more languages and themes in the future.
The puzzle generation process begins by creating a solution matrix, followed by the generation of clues and red herrings. The system utilizes 14 distinct clue types and 8 types of red herrings, ensuring that each generated puzzle has a unique solution. When translating linguistic components, the team prioritized correctness, clarity, naturalness, ease of generation, consistency, and diversity across languages.
For evaluating LLM performance, the study tested two models: o3-mini, which is characterized as a reasoning model, and GPT-4o mini, a non-reasoning model. The findings indicated that puzzle sizes 2×3 and 4×5 were appropriately challenging for GPT-4o mini and o3-mini, respectively. A notable discovery was the significant impact of red herrings: including five red herrings decreased o3-mini’s puzzle-level accuracy on 4×5 puzzles by 15±7%. This highlights red herrings as an effective mechanism for increasing difficulty for reasoning-focused models. Interestingly, o3-mini’s performance on 4×5 puzzles was not significantly affected by whether the puzzles were in English or Danish, or by the theme (a common ‘houses’ theme versus a culture-specific ‘smørrebrød’ theme). This suggests that advanced LLMs might possess logical reasoning abilities that generalize well across different languages and cultural contexts.
The research also investigated the influence of various clue types on difficulty but found no clear correlation between specific clue types and puzzle difficulty when the number of red herrings was kept constant. However, the puzzle generation algorithm naturally favors more informative clue types.
Also Read:
- DecompSR: Unpacking How Language Models Reason About Space
- Indian Riddles Uncover LLM Overconfidence and Multilingual Reasoning Gaps
In summary, MultiZebraLogic offers a valuable new resource for benchmarking the multilingual logical reasoning capabilities of LLMs. The datasets, which include 128 puzzles for training and 1024 for testing in each of the nine languages for sizes 2×3 and 4×5, are publicly available. The study underscores the effectiveness of red herrings in challenging reasoning models and demonstrates the potential for logical reasoning skills to generalize across diverse linguistic and cultural settings. For a deeper dive into the methodology and results, you can access the full research paper here.
