TLDR: Researchers introduce BesiegeField, a testbed based on the game Besiege, to evaluate if large language models (LLMs) can design complex machines from standardized parts. They benchmark LLMs using agentic workflows and reinforcement learning, finding that while challenging, LLMs can create functional designs, with performance improving through iterative feedback and structured reasoning. The study highlights the importance of spatial reasoning and high-level planning for LLMs in machine design.
A groundbreaking study explores whether large language models (LLMs) can move beyond understanding and generating text to actually creating complex machines. This research delves into “compositional machine design,” a task where machines are built from standard parts to fulfill specific functions, like moving or manipulating objects, within a simulated physical world.
To facilitate this investigation, the researchers introduced BesiegeField, a novel testing environment. Built upon the popular machine-building game Besiege, BesiegeField allows for the construction of machines using individual parts, simulates their physical behavior, and evaluates their performance based on defined rewards. The primary goal is to see if LLMs, given a set of mechanical parts and a way to measure success, can discover diverse and effective ways to combine these parts to achieve a desired function.
The motivation behind this research extends beyond artificial intelligence; it touches upon the future of engineering. Modern design processes are often lengthy and expensive, with valuable knowledge frequently lost or difficult to retrieve. An automated machine design system could streamline these challenges, acting as an “agent” capable of exploring vast design possibilities, leveraging existing knowledge, creating new designs for novel demands, and continuously improving through feedback.
The BesiegeField Environment
BesiegeField is a highly interactive and customizable platform. It leverages Besiege’s physics engine to simulate mechanical systems such as vehicles and catapults in various user-defined scenarios, complete with different terrains, obstacles, and external forces like wind and gravity. The environment offers nearly 80 types of building blocks, ranging from passive elements like logs to powered components like cogs and wheels. Machines are assembled by attaching new parts to existing ones, forming a structured “construction tree.” Crucially, the environment records detailed state information during simulations, providing rich feedback for the LLM agents.
This testbed strikes a unique balance: it incorporates realistic geometry and physics, uses parts with clear functional meanings, and follows simple compositional rules. Unlike simpler block-stacking games or overly complex CAD modeling, BesiegeField provides a tractable yet challenging environment to study creative and geometric reasoning in machine design. It even supports machine destruction, allowing for analysis of durability and failure modes.
Benchmarking LLMs: Cars and Catapults
To evaluate LLM capabilities, the study focused on two representative tasks: building a “car” and a “catapult.” The car task primarily tests static reasoning—ensuring correct part orientations, symmetry, and stability. The catapult task, on the other hand, challenges models with dynamic reasoning, requiring parts to coordinate over time to produce specific mechanical effects, like launching a boulder a long distance. Performance was measured by factors like moving distance for cars and throwing distance and height for catapults.
The researchers found that representing machines using a “construction tree” (which details how parts are attached relative to each other) was significantly more effective for LLMs than a simple coordinate-only representation. This highlights the importance of explicit structural information for LLM understanding.
Agentic Workflows for Design
The study explored several approaches for LLM agents to tackle machine design:
-
Single-Agent Setting: A single LLM agent generates a plan and then translates it into a machine design.
-
Iterative Editing: This more sophisticated workflow involves multiple agents: a “designer” creates an initial draft, an “inspector” critiques it, and a “refiner” proposes revisions. An “environment querier” runs simulations and summarizes feedback. This iterative loop, often guided by search strategies like Monte Carlo Tree Search, allows for continuous improvement.
-
Hierarchical Construction: Inspired by human design, a “meta-designer” agent first creates a high-level blueprint of major functional blocks (e.g., a suspension system) and their connections. Then, parallel “builder” agents construct the machine block by block, following this blueprint.
Key Findings and Challenges
The research revealed that compositional machine design is a demanding task for LLMs. While models like Gemini 2.5 Pro could consistently build functional machines, many struggled. Common failure patterns included incorrect part orientations, improper placements, failures to follow instructions, and flawed high-level reasoning. The study observed that providing environment feedback significantly improved machine performance, and maintaining an “edit history” helped reduce design errors.
Interestingly, the ability of LLMs to generate a coherent “chain-of-thought” (CoT) for design was crucial. When other LLMs were given CoT generated by the best-performing model, their designs improved, suggesting that high-level semantic reasoning is a major bottleneck. There was also a noticeable “misalignment” between an LLM’s conceptual CoT blueprint and the actual machine it generated.
Reinforcement Learning for Machine Design
Recognizing that simply scaling agentic systems might not be efficient, the researchers also investigated reinforcement learning (RL) to help LLMs internalize design experience. They curated a “cold-start” dataset of nearly 10,000 machine designs and their corresponding CoTs from Gemini 2.5 Pro. This dataset was used to finetune open-source LLMs, teaching them to align their reasoning with expert designs.
RL finetuning generally improved machine performance, primarily by increasing the proportion of valid and functional designs. While models tended to make detailed adjustments rather than exploring entirely new design strategies, the process showed promise in enhancing their ability to create precise configurations for smooth part coordination.
Also Read:
- Ensuring Robotic Safety: A Multi-Level Approach for LLM-Powered Agents
- Phantom Recall: The Hidden Flaw in Large Language Models’ Reasoning
Future Directions
This work highlights that compositional machine design demands precise spatial placement, accurate translation of high-level plans into detailed geometry, and a blend of conceptual and detailed reasoning. While this study focused on pure LLM-based reasoning, integrating multimodal information (text with visual or schematic representations) is identified as an important future direction. The need for RL methods that encourage diverse and novel design strategies, rather than converging on a narrow set of solutions, also remains a key challenge for advancing LLMs in this complex engineering domain.
