TLDR: A new research framework, featuring the TurtleSoup-Bench benchmark and Mosaic-Agent, evaluates Large Language Models’ (LLMs) imaginative reasoning in information-scarce environments. Unlike traditional benchmarks, this approach uses interactive ‘Turtle Soup’ puzzles, where LLMs must ask yes/no questions to uncover hidden stories. Experiments reveal that current LLMs struggle significantly compared to humans, exhibiting specific failure patterns like semantic fixation and deductive pruning issues, highlighting a need for more advanced exploratory capabilities.
Large Language Models (LLMs) are becoming increasingly central to autonomous agents, enabling advanced reasoning and decision-making. However, their capabilities are often tested in environments where all information is readily available and rules are clearly defined. Many real-world scenarios, like an archaeologist inferring daily life from limited artifacts or a police officer reconstructing a crime from sparse clues, demand a different kind of intelligence: imaginative reasoning. This involves proactively building, testing, and revising hypotheses in situations with incomplete information.
Existing benchmarks for LLMs often fall short in evaluating this dynamic, exploratory reasoning. They tend to focus on static question-answering or social deduction games, which don’t capture the iterative process of generating and testing ideas. To address this crucial gap, researchers have introduced a comprehensive framework centered around the classic “Turtle Soup” game.
Introducing TurtleSoup-Bench
The core of this new framework is TurtleSoup-Bench, the first large-scale, bilingual, and interactive benchmark specifically designed for imaginative reasoning. It comprises 800 unique “turtle soup” puzzles. In these puzzles, a solver is given only a brief, enigmatic scenario (the “soup surface”) and must uncover the complete underlying story (the “soup bottom”) by asking a series of yes/no questions. This process naturally requires an iterative loop of abductive (forming hypotheses) and deductive (testing hypotheses) logic.
The puzzles in TurtleSoup-Bench are sourced from both online communities and expert authors, ensuring a diverse and challenging set of scenarios across six narrative genres: Crime Thriller, Mind Game, Supernatural, Constant Change, Clever Logic, and Original (expert-authored) stories. Each scenario includes a “Key Clue Library” – expert-defined pivotal turning points that guide the reasoning process.
The Mosaic-Agent Framework
To assess LLMs’ performance in this interactive setting, the researchers propose the Mosaic-Agent, a novel multi-agent framework. It simulates the Turtle Soup puzzle-solving process through dynamic interaction between two main components: a Questioner agent and a Responder agent, supported by a Memory module.
The **Questioner Agent** acts like the human player. It employs a deliberative cognitive architecture inspired by human problem-solving, involving three processes:
- **Deliberation Agent**: This is the analytical core. It processes information from recent interactions and periodically conducts a global deliberation to update its internal “Belief State” (understanding of the story’s logic, details, and conclusion). It then identifies logical gaps and proposes potential questions.
- **Meta-cognition Agent**: This agent dynamically adjusts the overall strategy by classifying the puzzle’s narrative genre. It uses a “Smoothed Confidence” mechanism to prevent frequent strategy changes, ensuring a stable approach. Different genres have unique questioning strategies designed to target their typical logical structures.
- **Action Formulation Agent**: This component combines the analysis from the Deliberation agent with the chosen strategy from the Meta-cognition agent to generate and select the single best question to ask next, avoiding redundancy and maximizing information gain.
The **Responder Agent** acts as the “God” or the interactive environment. It’s designed to be deterministic and truthful. Given the Questioner’s question, it provides a standardized answer: “Yes,” “No,” or “Unknown.” Crucially, it also identifies if the question relates to a “Key Clue” from the puzzle’s solution, flagging such answers to guide the Questioner.
The **Memory Module** serves as a central hub, recording the complete interaction history (all questions and answers) and a curated record of only the “Key Clues” identified by the Responder. This allows the Questioner to reflect on the conversation and quickly access pivotal information.
Automated Evaluation Protocol
To objectively evaluate the LLMs’ performance, the framework uses a multi-dimensional evaluation protocol. It decomposes the true solution (soup bottom) into “Core Logic Points” and “Key Details.” The Questioner’s final summary is then assessed across three metrics:
- **Logic Accuracy**: Measures the coherence of the causal chain.
- **Detail Fidelity**: Measures the factual grounding.
- **Conclusion Match**: Provides a holistic assessment of the final summary against the ground truth.
These metrics are combined into an “Overall Score.” A human performance baseline, derived from expert players, is used as a reference to highlight the gap between LLM agents and human capabilities.
Experimental Findings and Limitations
Experiments with state-of-the-art LLMs, including Claude-3.7-Sonnet, Gemini-2.5-Flash, Deepseek-R1, GPT-4o, Qwen3-32B, and Llama3-8B-Instruct, revealed significant insights. Top-tier proprietary models generally outperformed open-source models, but a substantial performance gap remains between even the best LLMs and human experts (approximately 13 percentage points). This suggests that LLMs currently lack the highly effective intuition, creative hypothesis generation, and efficient elimination of irrelevant options that humans exhibit.
The study also found that model performance correlates strongly with the narrative genre, indicating that current LLM imagination might be a collection of specialized skills rather than a general capability. Furthermore, a systematic performance decline was observed on the English dataset compared to the Chinese one, suggesting that cultural and linguistic subtleties, along with potential ambiguity introduced during translation, increase the difficulty of reasoning.
Qualitative analysis identified four common failure patterns in LLMs:
- **Semantic Fixation**: Rigidly interpreting words literally, ignoring context.
- **Context Construction Failure**: Failing to integrate fragmented clues into a coherent global understanding.
- **Logic Blind Spots**: Struggling to conceive atypical causality or motivations outside common training data patterns.
- **Deductive Pruning Failure**: Ineffectively using negative feedback to eliminate incorrect hypotheses, leading to redundant exploration.
An ablation study confirmed the necessity of each component of the Mosaic-Agent, particularly highlighting the critical role of the “Key Clue” mechanism. Without this high-signal feedback, the agent’s exploration degrades significantly, underscoring that quality environmental feedback is crucial for efficient reasoning.
Also Read:
- PRELUDE: A New Benchmark Uncovers Gaps in AI’s Long-Context Reasoning
- Assessing Emotional Intelligence in Large Language Models: Introducing MME-Emotion
Conclusion
This research introduces a robust framework for evaluating the imaginative reasoning of LLMs, moving beyond static outcomes to assess the dynamic process of inquiry. The findings underscore the current limitations of LLMs in information-scarce environments and complex imaginative tasks, paving the way for future research focused on enhancing exploratory agent behavior and bridging the performance gap with human cognition.
