TLDR: This research paper evaluates the ability of Large Language Models (LLMs) to perform robotic task planning compared to classical planners like Fast Downward. The study assesses LLMs’ success rates and execution fidelity across various PDDL domains. Findings indicate that while LLMs perform well on simpler tasks and show improvement with reasoning capabilities, they struggle with complex scenarios requiring precise resource management and constraint compliance. The paper concludes that LLMs are not yet robust enough to replace classical planners entirely for real-world robotics and suggests hybrid approaches combining LLM flexibility with symbolic verification for future development.
Robotics relies heavily on task planning, which allows autonomous systems to decide on sequences of actions, manage resources, and adapt to changing situations. Traditionally, this has been handled by symbolic planners, such as Fast Downward, which use a formal language called Planning Domain Definition Language (PDDL) to create precise and verifiable solutions. While these classical planners are reliable, they can struggle with the complexities and uncertainties of real-world environments, often requiring detailed and time-consuming setup.
In recent years, Large Language Models (LLMs) like GPT-4, Claude, and Llama have shown remarkable abilities in common-sense reasoning and generating structured, multi-step outputs. This has led to significant interest in their potential for robotic task planning. LLMs can produce plans for simpler tasks by leveraging the vast amount of text data they were trained on. However, a major question remains: can these powerful generative models truly replace classical planning methods, especially when it comes to reliability and the ability to execute plans in real-world scenarios?
A recent study, titled “CAN LLM-REASONING MODELS REPLACE CLASSICAL PLANNING? A BENCHMARK STUDY” by Kai Goebel and Patrik Zips, delves into this very question. The researchers conducted a systematic evaluation of various state-of-the-art language models, directly prompting them with PDDL domain and problem files, and compared their planning performance against the Fast Downward planner across a range of benchmarks. You can find the full research paper here: Research Paper.
Evaluating LLMs Against Classical Planners
The study didn’t just measure how often LLMs succeeded in generating a plan; it also assessed how accurately those generated plans could be translated into executable actions. This “execution fidelity” is crucial, as an incorrect or unexecutable action in a robotic system can lead to safety risks or task failures. The researchers identified both the strengths and limitations of using LLMs in this context.
The evaluation used Fast Downward as a robust classical baseline, known for its efficiency and formal verification. Nine different LLMs were tested, including various versions of Claude Sonnet, Gemini Flash, Llama, and GPT models. These models were chosen to represent a snapshot of current advanced LLMs, with their reasoning capabilities assessed using benchmarks like GPQA Diamond Accuracy.
The LLMs were prompted to act as planning assistants, providing a high-level reasoning overview and then a step-by-step sequence of actions in a structured JSON format, complete with justifications for each action. This structured output was then evaluated across five distinct PDDL domains: Barman, Blocks, Elevator, Satellite, and Tidybot. These domains represent different levels of complexity in resource management, concurrency, and object handling, from stacking blocks to coordinating satellites and cleaning a household.
Key Findings: Strengths and Struggles
The results showed that while LLMs have made progress, particularly with models incorporating explicit reasoning mechanisms like Claude Sonnet 3.7 Thinking and GPT-o1, they still face significant challenges. These top-performing LLMs achieved a success rate of 63.4%, which is notable but still falls short of Fast Downward’s near-perfect success rate (97.85%).
When it came to execution fidelity – how many planned actions could actually be executed – Fast Downward perfectly executed all its planned actions. Among LLMs, GPT-o1 showed the highest execution fidelity at 73.4%, followed by Llama DeepSeek R1 and Claude Sonnet 3.5. This indicates that even when LLMs generate a plan, a significant portion of it might not be practically executable due to violations of domain constraints or incomplete action sequences.
The study also looked at planning time. Models with explicit reasoning, such as Claude Sonnet 3.7 Thinking and GPT-o1, generally took longer to generate plans. However, longer planning times did not always correlate with better success rates, suggesting that computational overhead doesn’t guarantee superior plan quality.
Performance varied significantly across different planning domains. LLMs performed well in simpler domains like “blocks” and “elevator,” likely because these problem structures align with patterns seen during their training. However, they struggled considerably with more complex domains like “barman” and “tidybot,” which demand precise resource management, consistent state tracking, and strict constraint compliance. This raises questions about whether success in simpler domains comes from genuine reasoning or merely recalling pre-trained solutions.
Also Read:
- Enhancing AI Agents with Graph Structures: A Comprehensive Overview
- CoEx: Enhancing LLM Agents Through Adaptive Planning and World Modeling
The Path Forward: Hybrid Approaches
The findings underscore a critical point: while LLMs show considerable promise for task planning, their current limitations in reliability and executability make them unsuitable as standalone general planners for real-world robotic applications. Execution failures, even partial ones, highlight the difficulty LLMs have in ensuring action validity over long planning horizons.
The researchers suggest that future work should focus on “hybrid approaches” that integrate LLMs with classical planning techniques. This synergy would combine the generative flexibility and common-sense reasoning of LLMs with the rigorous state-space search and correctness guarantees of symbolic planners. Such an arrangement could enhance the reliability and scalability of planning in autonomous robotics, especially in dynamic and uncertain environments. Iterative plan refinement, where LLMs continuously adjust plans based on execution feedback, is also highlighted as a promising direction, mimicking how humans approach complex tasks.
In conclusion, while LLM-based planning is still in its early stages, its rapid progress suggests a future where data-driven reasoning blends seamlessly with structured robotic task execution, leading to more adaptive and robust autonomous systems.
