TLDR: DecompSR is a new, large-scale dataset and framework designed to rigorously evaluate the compositional spatial reasoning abilities of Large Language Models (LLMs). It allows researchers to independently vary aspects like reasoning depth, language, entity names, distractors, and information order. Benchmarking experiments using DecompSR reveal that LLMs struggle with applying learned rules to more complex problems (productivity) and interpreting novel linguistic elements systematically. They also show tendencies to overgeneralize based on input order or the presence of noise. However, LLMs demonstrate robustness to variations in entity names and different natural languages. The dataset’s ‘correct by construction’ nature ensures that observed model failures are due to reasoning limitations, not data flaws.
Large Language Models (LLMs) are becoming increasingly capable of solving complex problems, but understanding how they truly reason remains a significant challenge. Are they genuinely applying rules and components in new situations, or are they just good at recognizing patterns? This question is at the heart of a new research paper that introduces DecompSR, a massive dataset and framework designed to dissect the compositional reasoning abilities of LLMs.
The paper, titled DecompSR: A dataset for decomposed analyses of compositional multihop spatial reasoning, highlights a critical gap in current LLM evaluation methods. Many existing benchmarks focus solely on whether an LLM gets the final answer right, which can hide underlying weaknesses. Models might use ‘shortcut learning’ by picking up on statistical cues in the data rather than truly understanding and combining information. This often leads to models that perform well on familiar tasks but fail when faced with new problems requiring genuine rule-based generalization.
DecompSR, which stands for Decomposed Spatial Reasoning, aims to fix this by providing a rigorous and provably correct benchmark. It’s a large dataset with over 5 million data points, built procedurally to ensure its correctness. This means every problem instance is logically sound and has a unique, deducible answer, verified by a symbolic solver. This ensures that any observed failures in LLM performance are due to the model’s reasoning capabilities, not flaws in the dataset itself.
How DecompSR Works
The core idea behind DecompSR is to systematically vary different aspects of compositional reasoning. Each instance in the dataset is a triple: a ‘story’ (s), a ‘question’ (q), and an ‘answer’ (a). The story describes spatial relations between entities, and the question asks for the relation between two specific entities, often requiring multiple steps of inference. The framework allows researchers to independently control several parameters:
- Reasoning Depth (k): This controls the number of steps an LLM needs to chain together to find the answer. By varying ‘k’ from 1 to 100, DecompSR tests an LLM’s ‘productivity’ – its ability to apply learned rules to increasingly complex problems.
- Language Variation: Stories and questions can be generated in different natural languages (English, Swedish, Hindi) or with synthetic English where directional words are replaced by made-up ‘nonce words’. This tests ‘systematicity’ (can the model learn new symbols as replacements for known relations?) and ‘linguistic substitutivity’ (is reasoning robust to changes in language?).
- Entity Representation (Node Names): The names of objects in the story can be symbolic, common human names, city names, or randomly generated nonce words. This probes ‘substitutivity’ – whether the model’s reasoning is affected by superficial changes in entity labels.
- Presence of Distractors (Noise): Irrelevant sentences can be added to the story. This tests an LLM’s ability to focus on relevant information and avoid ‘overgeneralization’ by being disrupted by extraneous details.
- Information Order (Story Order): The sequence of sentences in the story can be ordered or shuffled. This investigates ‘overgeneralization’ related to order-dependent heuristics and the robustness of the model’s internal representation of the spatial scene.
Key Findings from Benchmarking LLMs
The researchers conducted extensive benchmarking experiments on a range of state-of-the-art LLMs, including models from OpenAI, Anthropic, Moonshot AI, and others. The results revealed several critical insights:
- Productivity Challenges: LLMs generally struggle with ‘productivity’. As the number of reasoning steps (k) increased, the accuracy of models significantly dropped, often approaching a random guess rate for higher ‘k’ values. This suggests that current LLMs do not achieve strong productive generalization in spatial reasoning tasks.
- Systematicity Weaknesses: When tested with ‘nonsense words’ for directions, LLMs showed a drastic drop in performance, collapsing to the guess rate for even simple multi-step problems. This indicates a significant lack of ‘systematicity’ – the ability to consistently interpret and use novel symbols.
- Overgeneralization Tendencies: LLMs exhibited ‘overgeneralization’ when stories were shuffled or contained distracting information. Their accuracy decreased when the input order was varied or when irrelevant sentences were present, suggesting they rely on order-based strategies or are easily disrupted by noise.
- Linguistic Robustness: On a more positive note, LLMs demonstrated a degree of robustness to ‘linguistic variation’. Their performance was relatively insensitive to changes in entity names (e.g., from symbolic to human names) and showed reasonable performance across different natural languages (English, Hindi, Swedish), although some language-specific differences were observed.
The paper also explored using LLMs to translate natural language stories into Answer Set Programming (ASP) facts for symbolic reasoning. While LLMs could perform this translation for smaller ‘k’ values, their ability to abstract information reliably deteriorated as the reasoning depth increased, highlighting brittleness in their understanding.
Also Read:
- SpatialTraceGen: Enhancing VLM Spatial Reasoning Through Verified Data Generation
- Ariadne: Expanding Vision-Language Model Reasoning in Spatial Tasks
Conclusion
DecompSR offers a powerful new tool for researchers to conduct fine-grained analyses of LLM compositional reasoning. By allowing precise control over various aspects of problem complexity and presentation, it moves beyond simple accuracy metrics to diagnose specific strengths and weaknesses. The findings suggest that while LLMs show some resilience to linguistic changes, they still face significant hurdles in systematic and productive generalization, especially in complex multi-hop spatial reasoning tasks.
