TLDR: A research paper titled “Stuck in the Matrix” investigates the spatial reasoning abilities of large language models (LLMs) using five text-based grid tasks (Quadrant, Transformation, Distance, Word Search, Slide). The study found that while LLMs perform moderately well on small grids, their accuracy drops rapidly (average 42.7% loss) as grid complexity and size increase, indicating a lack of robust spatial representations. Anthropic models generally outperformed OpenAI models. The findings highlight a significant gap in LLMs’ ability to handle geometric and spatial tasks from text, suggesting architectural limitations and avenues for future research in multimodal inputs and improved spatial data representations.
A new research paper titled “Stuck in the Matrix: Probing Spatial Reasoning in Large Language Models” delves into a critical limitation of current large language models (LLMs): their ability to understand and reason about spatial relationships when presented with text-based information. Authored by Maggie Bai, Ava Kim Cohen, Eleanor Koss, and Charlie Lichtenbaum, this study highlights a significant gap between an LLM’s impressive linguistic capabilities and its struggle with geometric and spatial tasks.
The inspiration for this research came from observing LLMs attempting to play simple grid-based games like 2048 or Connect 4. While the models grasped the game’s strategy, they frequently made invalid moves due to a failure to accurately interpret the board state. This issue wasn’t just about initial board reading but also applying principles like gravity in Connect 4, underscoring the challenges of text-based spatial reasoning for these models. While visual language models (VLMs) might address these issues, this paper specifically investigates LLMs’ performance with text-only inputs, which has implications for fields like finance where interpreting tabular data is crucial.
Testing Spatial Understanding with Five Core Tasks
To rigorously evaluate LLMs, the researchers designed a suite of five distinct tasks, each progressively increasing in complexity and targeting a specific aspect of spatial understanding. All tasks used simple symbols on square grids, presented as ASCII text:
Quadrant: Models had to identify which quadrant of a Cartesian grid contained a specific symbol (‘X’).
Transformation: This task required models to reflect an ‘X’ across a horizontal or vertical centerline and provide its new coordinates.
Distance: Models were presented with various letters and an ‘X’ on a grid, then asked to identify the closest and farthest letters relative to the ‘X’.
Word Search: LLMs were given a grid of letters and a target word, and their goal was to locate the word by providing its coordinates.
Slide: In this task, models had to “slide” an ‘X’ in a cardinal direction until it hit a ‘wall’ (#) or the grid’s edge, then output its final coordinates.
The study evaluated four prominent LLMs: GPT-4o, GPT-4.1, Claude 3.7 Sonnet (referred to as “No Thinking”), and Claude 3.7 with 16,000 thinking tokens (referred to as “Medium Thinking”). Each task was scaled by increasing grid dimensions, allowing the researchers to observe how performance changed with complexity.
Key Findings: A Rapid Decline in Performance
The results were consistent and revealing across all tasks and models: while LLMs showed moderate success on smaller grids, their performance deteriorated rapidly as grid sizes increased. On average, accuracy dropped by 42.7%, with some tasks seeing a decline as high as 84%. This suggests that LLMs struggle to generalize simple spatial reasoning to larger, more intricate configurations, hinting at a fundamental lack of robust spatial representations in their underlying architectures.
Specific observations included:
Quadrant and Transformation: Errors often stemmed from misreading the initial grid or difficulty establishing centerlines. OpenAI models performed particularly poorly on larger grids.
Distance: Models frequently made mathematical errors in calculating distances, indicating more than just positional misidentification.
Word Search: Despite being a language-based task, models struggled significantly, especially with longer words and non-traditional orientations (e.g., vertical or diagonal spelling). Hallucination, where models claimed to find words they hadn’t, was also observed.
Slide: This multi-step reasoning task proved very difficult, with accuracy dropping quickly as the number of required slides increased. Models often ignored walls or miscounted positions.
Interestingly, the Anthropic models (Claude 3.7 variants) consistently outperformed the OpenAI models (GPT-4o, GPT-4.1) across most tests, sometimes by as much as 30% in accuracy. GPT-4o, in particular, also exhibited a higher rate of parsing errors, failing to provide responses in a usable format as complexity grew.
Also Read:
- Unmasking LLM Reflection: Why Self-Correction Falls Short in Open-Ended Tasks
- Uncovering the Geometry of Thought: How Topology Reveals Quality in LLM Reasoning
Exploring Solutions: Tokenization and Future Directions
The researchers also experimented with different grid layouts and tokenizations to see if presentation format could mitigate the accuracy loss. For the Quadrant task, grids with “no spaces” or explicit “centerlines” showed significant improvement, especially for OpenAI models. This suggests that how text is tokenized and presented can impact an LLM’s ability to “count” and interpret grid elements. However, similar tokenization changes did not yield significant improvements for the more complex Word Search task, implying deeper issues beyond simple input formatting.
In conclusion, this paper underscores the current limitations of LLMs in spatial reasoning over text. While they excel at linguistic tasks, their architectural design appears to hinder their ability to form and maintain robust spatial representations. The authors suggest several avenues for future research, including exploring multimodal inputs (e.g., presenting grids as images), developing more efficient text-based representations of spatial data, improving LLMs’ counting and mathematical capabilities, and experimenting with different output methods like code generation. For a deeper dive into the methodology and results, you can read the full paper here.
