TLDR: BuildArena is the first physics-aligned interactive benchmark designed to evaluate large language models (LLMs) in engineering construction tasks. It features customizable tasks (Support, Transport, Lift) with varying difficulty, a 3D Spatial Geometric Computation Library for language-driven interaction, and an LLM agentic workflow. Experiments show LLMs can perform elementary construction and exhibit creative problem-solving, but struggle with complex assembly and high-precision tasks, with Grok-4 showing the strongest performance. The benchmark highlights current LLM limitations in physical reasoning and offers a foundation for future advancements in LLM-based construction automation.
A new research paper introduces BuildArena, a groundbreaking benchmark designed to evaluate how well large language models (LLMs) can handle engineering construction tasks. This interactive platform aims to bridge the gap between the advanced reasoning capabilities of LLMs and the complex, physics-constrained world of physical construction.
The field of engineering construction automation seeks to translate natural language instructions into functional, physically sound structures. This requires sophisticated reasoning and adherence to strict physical laws. While LLMs have shown impressive abilities in understanding language, mathematical reasoning, and even code generation, their practical skills in building physical objects have largely remained untested until now.
Introducing BuildArena
Developed by Tian Xia, Tianrun Gao, Wenhao Deng, Long Wei, Xiaowei Qian, Yixian Jiang, Chenglei Yu, and Tailin Wu from Westlake University, BuildArena is the first benchmark of its kind. It allows researchers to comprehensively assess LLMs’ capacity for language-driven and physics-grounded construction automation. The project page can be found at arxiv.org/pdf/2510.16559.
BuildArena offers several key contributions: a highly customizable framework for in-depth analysis of LLMs, an extendable task design strategy covering various mechanics and difficulty levels, a 3D Spatial Geometric Computation Library to support construction based on language instructions, and a baseline LLM agentic workflow for evaluating diverse model capabilities.
How BuildArena Works
The benchmark operates through a three-part methodology: task definition, LLM-based construction, and simulation-based evaluation. Tasks are designed with varying levels of difficulty across six dimensions: quantification, robustness, magnitude, compositionality, precision, and ambiguity. There are three main task categories:
- Support: Involves building static structures like bridges to support a load across a gap.
- Transport: Focuses on constructing mechanical structures capable of directional movement, such as vehicles carrying cargo.
- Lift: Requires building rockets or rocket-powered aircraft capable of vertical movement.
To enable LLMs to interact with a physical environment, BuildArena uses a custom-developed 3D Spatial Geometric Computation Library. This library mirrors the logic of the popular physics sandbox game Besiege, allowing LLMs to issue commands (Build, Refine, Query, Control) through a language interface. This ensures that actions taken by LLMs in the language space have consistent physical effects in the simulated environment.
The LLM agentic workflow is structured to facilitate complex construction. It involves five collaborative LLM entities: a Planner that outlines the construction strategy, a Drafter that creates detailed schematics, a Reviewer that checks for structural and functional feasibility, a Builder that executes the construction steps using the Spatial Geometric Computation Library, and a Guidance agent that provides step-by-step instructions. For transport tasks, a Controller is also used to design control configurations and sequences.
Evaluation and Key Findings
BuildArena evaluates LLMs based on performance metrics like the number of parts used (fewer is better), success rate, and task-specific indicators (e.g., maximum transport distance, maximum load weight, maximum height). Cost metrics, such as input/output tokens and LLM requests, are also tracked.
Experiments with eight frontier LLMs (GPT-4o, Claude-4, Grok-4, Gemini-2.0, DeepSeek-3.1, Qwen-3, Kimi-K2, and Seed-1.6) revealed several insights. BuildArena proved effective in testing LLM construction capabilities, with performance generally decreasing as task difficulty increased. The LLMs demonstrated elementary construction skills, adapting to challenges like scaling up components for larger payloads and even proposing unconventional solutions, such as propulsion-powered carriers or wheel-integrated bridges. They also constructed structures mirroring real-world engineering practices, suggesting an implicit understanding of spatial information from text.
However, significant limitations were also observed. LLMs struggled with hierarchical assembly tasks, showing a sharp drop in success rates as complexity increased. High-precision tasks with low robustness, like the Lift tasks, also posed a major challenge, with most models failing at higher difficulty levels. The most common failure reason was spatial conflict, indicating difficulty in accurately tracking updated spatial structures.
Among the models, Grok-4 exhibited the strongest overall performance, particularly in precision and robustness. Other models showed similar strengths in handling magnitude and ambiguity, but consistent weaknesses in quantification, robustness, compositionality, and precision. Interestingly, the study found that massive inference (high token usage) did not guarantee high performance, with many successful constructions achieved with moderate token consumption.
Also Read:
- Unlocking 6D Spatial Reasoning in AI: A New Benchmark for Multimodal Models
- Enhancing VLM Agent Intelligence Through Explicit World Model Reasoning
Future Directions
While BuildArena marks a significant step, the researchers acknowledge limitations. The framework currently lacks an extended loop for refining construction results based on simulation feedback, which could further unlock LLM potential. Additionally, the diversity of basic units in the module library is limited, suggesting a need for community contributions to enrich the available assets. BuildArena paves the way for future research in enabling and evaluating LLMs for complex engineering constructions, bringing them closer to interacting with the physical world.
