PlanQA: Uncovering How Language Models Navigate Spatial Challenges in Digital Floor Plans

TLDR: PlanQA is a new benchmark evaluating Large Language Models’ (LLMs) spatial reasoning using structured 2D floor plan data (JSON/XML). It tests metric, topological, geometric planning, visual logic, and rule-based reasoning. Findings show LLMs excel at basic spatial queries but struggle with complex tasks like planning and constraint validation, especially in cluttered layouts. The paper suggests integrating external solvers or improving training with explicit spatial constraints to enhance LLMs’ real-world spatial understanding.

Large Language Models (LLMs) have shown impressive abilities in understanding and generating text, but their grasp of real-world spatial and geometric reasoning has remained a significant challenge. A new research paper introduces PlanQA, a diagnostic benchmark designed to rigorously evaluate how well LLMs can reason about 2D indoor environments, like kitchens, living rooms, and bedrooms, when these spaces are described using structured text formats such as JSON or XML.

The core idea behind PlanQA is to test LLMs on their direct, unaided reasoning capabilities, without relying on external tools or visual input. This means the models are given a text-based description of a floor plan, including object positions, sizes, and orientations, and then asked questions that require spatial inference.

What Does PlanQA Evaluate?

PlanQA goes beyond simple queries. It includes a diverse set of 18,000 questions across 1,800 unique 2D layouts. These questions are categorized into several types to test different aspects of spatial understanding:

• Metric Reasoning: Questions about exact measurements, such as distances between objects or the area of a specific space.
• Topological Reasoning: Understanding relationships like adjacency or containment.
• Geometric Planning: Complex tasks involving multiple steps, such as determining if a new object can fit into a space without overlapping existing items, or finding the shortest path through a room while maintaining clearance.
• Visual Logic: Tasks related to visibility and occlusion, like whether one object is visible from another point.
• Rule-Based Validation: Checking if a layout adheres to design constraints, such as proper clearance around furniture or the presence of essential items.

The benchmark uses a structured evaluation protocol, comparing model answers against precise, deterministically derived ground truths. For numerical answers, a small tolerance (e.g., 2% relative error) is allowed, while boolean or list-based answers require exact matches or geometric plausibility checks.

Key Findings from LLM Evaluations

The researchers evaluated 11 different LLMs, including well-known models like GPT-4.1, DeepSeek-R1, and Llama-3.3-70B-Instruct. The results revealed a clear “blind spot” in current LLMs regarding consistent spatial reasoning:

• Models performed relatively well on basic metric (distance, area) and visual logic (line-of-sight) queries, often achieving over 95% accuracy.
• However, performance significantly dropped for more complex tasks like geometric planning, topological relations, and rule-based validation, with accuracy often falling below 50%. This indicates that LLMs struggle to simulate physical constraints or maintain spatial coherence.
• While larger models generally performed better, the gains in accuracy plateaued beyond approximately 30 billion parameters, suggesting that simply increasing model size isn’t enough to solve these complex spatial problems.
• The type of room layout also influenced performance. Kitchens, with their more regular, grid-like arrangements, yielded higher accuracy compared to living rooms and bedrooms, which often have denser, more cluttered, and organically arranged furniture.
• An ablation study showed that changing the layout encoding from JSON to XML had little impact on model performance, suggesting that the models are somewhat robust to low-level representation details.

Also Read:

• Unlocking Deeper AI Understanding of Human Videos with HV-MMBench
• New Benchmark Reveals Gaps in AI’s Physics Reasoning

The Path Forward

The PlanQA benchmark highlights that current LLMs do not form sufficiently robust internal representations for complex spatial inference. The authors suggest two main directions for future work:

1. Integrating External Solvers: In the short term, LLMs could be paired with external geometric solvers or symbolic planning modules to handle tasks like collision avoidance and clearance reasoning more accurately.
2. Improved Training Regimes: For the longer term, new training methods that explicitly incorporate spatial constraints, perhaps through specialized objectives or data augmentation with challenging, irregular layouts, could help LLMs develop a more intrinsic understanding of spatial properties.

PlanQA serves as a crucial diagnostic tool, inspiring further research into language models that can accurately infer and manipulate spatial and geometric properties in practical settings, from architectural design to robotics. You can find the full research paper here: PlanQA: A Benchmark for Spatial Reasoning in LLMs using Structured Representations.