TLDR: CoEx is a new hierarchical LLM agent architecture that addresses the limitations of static world models and exploitation bias in existing agents. It achieves this by co-evolving a dynamic neurosymbolic world model with subgoal-driven exploration. This framework, comprising a Planner, Actor, and Adaptive Belief State, allows agents to dynamically adapt plans and integrate new discoveries. Experiments show CoEx outperforms current LLM agent paradigms in complex planning and exploration tasks across various environments like ALFWorld, PDDL, and Jericho, while maintaining computational efficiency.
Large Language Models (LLMs) have significantly advanced the capabilities of AI agents, allowing them to perform complex tasks. However, existing LLM agents often struggle in new environments with limited prior information. This is primarily due to two main limitations: their reliance on a static internal world model, which can become misaligned with the real world, and an ‘exploitation bias’ that favors repeating known successful actions over exploring new possibilities.
Researchers Minsoo Kim and Seung-won Hwang from Seoul National University have introduced a novel hierarchical agent architecture called CoEx, short for Co-evolving World-model and Exploration. This new framework aims to overcome these limitations by allowing the LLM’s planning abilities to evolve alongside a dynamically updated model of the world. You can read the full research paper here: CoEx – Co-evolving World-model and Exploration.
How CoEx Works: A Hierarchical Approach
CoEx operates with a hierarchical design, separating the agent’s functions into three main components:
- The Planner: This is an LLM agent that works at a ‘subgoal-level’. Instead of planning individual, low-level actions, it reasons and plans abstract subgoals. This allows CoEx to strategically plan for exploration when faced with uncertainty. The Planner can also dynamically adjust its plan based on new information, a process called ‘dynamic replanning’.
- The Actor: This LLM agent is responsible for executing the specific subgoals generated by the Planner. It translates these subgoals into low-level actions to interact with the environment. The Actor is designed to efficiently complete its assigned subgoal and can self-terminate if the subgoal is achieved, deemed unachievable, or exceeds a set number of steps, passing control back to the Planner.
- The Adaptive Belief State: This is the core of CoEx’s ability to adapt its understanding of the world. It’s an explicit, adaptable representation that continuously updates as subgoals are planned and executed. It combines two complementary forms of memory:
- Symbolic Memory: This is a code-based, object-oriented representation that stores concrete, factual information like the agent’s location, object states, and visited locations. It’s designed for fast, efficient tracking of low-level state changes.
- Structured Textual Memory: This part captures the agent’s higher-level understanding, manages uncertainty, and synthesizes knowledge using natural language. It’s updated through a two-stage process of ‘verification’ (assessing subgoal execution outcomes) and ‘synthesis’ (generating structured updates to the belief state based on learned facts and observations).
This unified approach allows CoEx to achieve ‘co-evolving’ agent planning and adaptive world modeling, directly addressing the limitations of older, monolithic LLM agent designs.
Overcoming Limitations
CoEx directly tackles the ‘exploitation bias’ by instantiating exploratory behavior at the subgoal level. This means exploration isn’t dependent on action-level demonstrations but can be easily orchestrated as a standard subgoal for the Planner. Furthermore, by defining an explicit and adaptable belief state, CoEx overcomes the ‘limited adaptation’ issue. Unlike implicit world models in monolithic LLMs, CoEx’s belief state can be reliably updated in response to new experiences, including exploratory ones, allowing it to resolve uncertainty arising from partial observability.
Experimental Success
The researchers evaluated CoEx across a diverse set of challenging agent scenarios, including ALFWorld (text-based embodied tasks), PDDL (classical symbolic planning), and Jericho (text-based adventure games requiring exploration and world modeling). CoEx demonstrated strong performance, often outperforming existing state-of-the-art LLM agent baselines like ReAct, Reflexion, AdaPlanner, HiAgent, ExpeL, and WALL-E.
For instance, in ALFWorld, CoEx achieved a high total success rate, showcasing its ability to handle tasks requiring both planning and exploration. In PDDL domains, it significantly improved both success and progress rates, highlighting its capability in complex planning. On Jericho games, CoEx excelled in exploration, demonstrating substantial gains in success and progress rates, especially in more difficult scenarios.
The study also found that CoEx is computationally efficient, with its additional components incurring only minimal overhead. Most of the computational cost remains in the Actor, which is shared with other baseline methods. This efficiency is achieved because planning and belief updates occur at the subgoal level, not at every low-level action step.
Also Read:
- Enhancing AI Agents with Graph Structures: A Comprehensive Overview
- New Algorithm Enhances LLM Tool Use with Adaptive Learning
Evolving World Model
A key insight from the research is how CoEx’s world model continuously evolves. Through trajectory analysis, it was observed that the belief state integrates execution feedback, allowing the Planner to strategically adapt its subgoals based on newly discovered information about the environment. This dynamic adaptation helps the agent learn from errors and unexpected outcomes, leading to eventual task success.
In conclusion, CoEx presents a significant step forward in LLM agent design, offering a robust framework for agents to learn, explore, and adapt their understanding of the world in real-time, leading to more capable and flexible AI systems.
