TLDR: This research paper explores using large language models (LLMs), specifically GPT-4o, to automatically generate abstract planning domains and problems from detailed PDDL descriptions and natural language abstraction objectives. The study investigates four categories of abstraction: alternative concrete actions, sequences of concrete actions, action/predicate parameters, and combinations thereof. While GPT-4o shows promise in simpler abstraction tasks, its performance decreases with increasing complexity, highlighting both the potential and current limitations of LLMs in this challenging area of AI planning.
Artificial intelligence systems often face complex tasks that require them to make decisions and plan actions. To handle this complexity, humans naturally use abstraction – the ability to focus on important details while ignoring irrelevant ones. This crucial cognitive skill has inspired significant research in AI, particularly in areas like planning, where simplifying tasks can lead to more efficient and understandable systems.
A recent preliminary report, titled Using Large Language Models for Abstraction of Planning Domains – Extended Version, explores how large language models (LLMs) can be leveraged to automate this abstraction process for AI planning domains. The paper, authored by Bita Banihashemia, Megh Patel, and Yves Lespérance, investigates the use of in-context learning with LLMs to generate abstract planning domains and problem instances.
The Challenge of Abstraction in AI Planning
In AI, planning involves defining a sequence of actions to achieve a goal from an initial state. These planning problems are often described using languages like PDDL (Planning Domain Definition Language), which specifies the rules of a world (domain) and a particular scenario (problem instance). While powerful, concrete PDDL descriptions can be very detailed, making planning computationally expensive and difficult to interpret.
The goal of abstraction is to create a higher-level, simplified representation of a domain that still captures the essence of the problem relevant to a specific purpose. For example, instead of planning every step of booking a flight and a hotel, an abstract plan might simply involve “booking transportation” and “booking accommodation.” The challenge lies in automatically generating these useful abstractions that align with a given objective, as doing it manually is costly and time-consuming.
LLMs to the Rescue: Automating Abstraction
This research proposes using advanced LLMs, specifically GPT-4o, to tackle the Planning Domain Abstraction Generation (PDAG) task. Given a detailed (concrete) PDDL domain and problem, a natural language description of the domain, and a natural language statement of the abstraction’s purpose, the LLM is asked to generate a simplified (abstract) PDDL domain and problem.
The researchers explored three main categories of abstraction, plus a combination:
- Abstraction of Alternative Concrete Actions: This involves combining multiple ways to achieve a sub-goal into a single, higher-level action. For instance, `book_hotel` and `book_airbnb` could both become `book_accommodation`. This also means abstracting related types (like `hotel` and `airbnb` into `accommodation`) and predicates.
- Abstraction of Sequences of Concrete Actions: Here, a series of low-level actions that achieve a specific outcome are merged into one abstract action. An example would be abstracting `enter_username` followed by `enter_password` into a single `login` action.
- Abstraction of Action/Predicate Parameters: This category focuses on removing irrelevant details by eliminating specific parameters from actions or predicates. For example, if the “room view” (e.g., ocean view, garden view) is not important for a high-level booking plan, that parameter and associated information can be removed.
- Abstraction of Alternative Sequences of Concrete Actions: This combines the first two categories, abstracting multiple different sequences of actions into a single high-level action.
How the LLM Learns and is Evaluated
The study used in-context learning techniques, including zero-shot (no examples) and one-shot (one example) prompting, augmented with chain-of-thought reasoning (encouraging step-by-step thinking) and role-play (asking the LLM to act as a PDDL expert). The benchmark examples used were entirely new, ensuring the LLM had not been trained on them.
To evaluate the generated abstract domains and problems, a hybrid approach was used. Automated tools like VAL (for syntax checking) and the Fast Downward planner (to generate plans and detect errors) were employed. Crucially, human experts also reviewed the results, comparing them against sample abstract domains created by knowledge engineers. This human evaluation was vital because there can be multiple valid ways to abstract a domain, and the LLM might use different but equally correct terminology.
Also Read:
- Bridging Minds: How Machine Learning is Shaping the Future of Belief-Desire-Intention Agents
- AlphaOPT: A Self-Improving AI for Smarter Optimization Modeling
Key Findings and Future Outlook
The experiments showed that GPT-4o generally performed well in simpler abstraction settings, particularly for abstracting action/predicate parameters, often generating almost error-free results. However, as the complexity increased – for alternative concrete actions, sequences of actions, and especially alternative sequences of actions – the LLM made more mistakes and produced more syntax errors. Common issues included failing to minimize high-level predicates or incorrectly using PDDL keywords like ‘OR’.
Despite these challenges, the research highlights the significant potential of LLMs in automating the generation of planning domain abstractions. Future work aims to expand the types of examples, use more expressive PDDL fragments, build datasets for fine-tuning LLMs, and develop more sophisticated automated validation mechanisms to ensure the correctness of the generated abstractions. The ultimate goal is to enable LLMs to not only generate abstractions but also to formally verify their soundness and completeness.
