TLDR: CoPiC (Code Driven Planning with Domain-Adaptive Critic) is a new framework that improves how Large Language Models (LLMs) plan tasks for AI agents. It addresses the issues of high query costs and short-term planning by having LLMs generate flexible, high-level planning programs. A separate domain-adaptive critic then evaluates these programs to select the best plan for long-term rewards. CoPiC alternates between planning and learning phases, refining its programs and critic based on environmental feedback. It significantly outperforms existing LLM-based planning methods in success rate and reduces token costs across environments like ALFWorld, NetHack, and StarCraft II, demonstrating superior efficiency and adaptability.
Large Language Models, or LLMs, have become incredibly powerful tools for AI agents, especially when it comes to planning out complex tasks. They can break down high-level instructions into a series of steps, making them useful in various scenarios, from managing household chores in virtual environments to playing intricate games. However, these general-purpose models often struggle with the specific demands of a particular environment. Their broad knowledge can sometimes lead to plans that sound good but aren’t actually feasible in the real (or simulated) world.
Traditionally, to overcome this challenge, AI agents would frequently ask the LLM for feedback, refining their plans step-by-step based on immediate observations. While this approach improves plan accuracy, it comes with a significant drawback: high query costs. Each interaction with the LLM incurs a cost, and frequent queries can quickly become expensive. Moreover, relying solely on immediate feedback can prevent the LLM from developing plans that consider long-term goals and rewards, leading to less efficient or suboptimal strategies.
Introducing CoPiC: A Smarter Way to Plan
To address these limitations, researchers have proposed a novel framework called Code Driven Planning with Domain-Adaptive Critic, or CoPiC. This approach shifts away from constant LLM queries for plan refinement. Instead, CoPiC leverages LLMs to generate a diverse set of high-level planning programs. Think of these programs as flexible blueprints that can iteratively create and adjust candidate plans.
The core of CoPiC lies in its two main components: an LLM planner and a domain-adaptive critic. The LLM planner is responsible for generating these high-level planning programs. These programs are designed to interact with the environment and dynamically produce and refine candidate plans. This significantly reduces the need for frequent, costly LLM queries that would otherwise be required for static, step-by-step plan generation.
Once candidate plans are generated, the domain-adaptive critic steps in. This critic is a trained module that evaluates these candidates, selecting the plan that is most aligned with long-term rewards. By integrating the efficiency of planning programs with the effectiveness of a critic that understands the specific domain, CoPiC not only cuts down on LLM query costs but also enhances the overall quality of the plans.
How CoPiC Learns and Adapts
CoPiC operates in an alternating cycle of a Planning Phase and a Learning Phase. In the Planning Phase, the LLM planner generates its programs, which then produce candidate plans. The domain-adaptive critic evaluates these plans and selects the best one for execution. The chosen plan is then carried out in the environment, leading to a new observation and a reward.
The Learning Phase is where CoPiC truly shines in its adaptability. The results from the execution of plans, including observations, actions, and rewards, are used to refine the planning programs themselves. This is done through a process called “Programs Evolution with History Summarization,” where the LLM analyzes past interactions, especially failures, to debug and improve its program generation. Simultaneously, the critic module is fine-tuned using a reinforcement learning framework, specifically Proximal Policy Optimization (PPO) with Low-Rank Adaptation (LoRA). This ensures the critic continuously learns from domain experience and becomes better at selecting plans that lead to long-term success.
Impressive Performance Across Diverse Environments
CoPiC was rigorously tested in three challenging environments: ALFWorld (a household task simulation), NetHack (a complex roguelike game), and StarCraft II Unit Building (a real-time strategy game focusing on resource management and unit production). The results were remarkable. CoPiC consistently outperformed advanced LLM-based baselines, demonstrating its superior planning capabilities and efficiency.
On average, CoPiC achieved a 20.29% improvement in success rate and a substantial 79.39% reduction in token costs compared to existing methods. For instance, in ALFWorld, it showed a 20.40% increase in success rate and an 83.76% reduction in cost. In NetHack, particularly for complex tasks like “Upgrade Exp Level to 3,” CoPiC reduced LLM queries by 96.11% while improving the success rate by 10%, by prioritizing long-term strategies. In StarCraft II Unit Building, it delivered a 44.75% increase in success rate and an 87.86% reduction in cost on hard tasks.
A key advantage of CoPiC is its ability to generalize effectively to unseen tasks without additional LLM querying costs or critic fine-tuning during testing. This “zero-shot adaptation” capability is a direct result of its integrated planning programs and domain-adaptive critic.
The Importance of Evolution and Evaluation
Ablation studies further highlighted the critical roles of CoPiC’s components. The “Programs Evolution” module was shown to iteratively refine planning programs, significantly boosting performance. For example, in ALFWorld tasks, the average success rate improved from 75.60% initially to 100% after several iterations of program evolution. The “Critic Scoring” module also proved essential; without it, and with random plan selection, the success rate was consistently 20% to 60% lower, even with the same interaction costs. This confirms that the critic is vital for selecting high-quality, long-term rewarding plans.
Furthermore, CoPiC demonstrated compatibility with various open-source LLMs, such as DeepSeek and Qwen2.5-14B, showing its robustness beyond proprietary models.
Also Read:
- Foundation Models Navigate Virtual Worlds: New Strategies for Reinforcement Learning
- MCTS-EP: Enhancing AI Agents for Complex Real-World Tasks
Looking Ahead
CoPiC represents a significant step forward in enabling LLMs to act as more efficient and effective planners for AI agents in complex, dynamic environments. By generating flexible planning programs and employing a domain-adaptive critic, it addresses the challenges of high query costs and short-sighted planning. The researchers are committed to expanding CoPiC’s application to even more intricate tasks, including full games like StarCraft II and Civilization, and more complex real-world scenarios. You can read the full research paper here: Code Driven Planning with Domain-Adaptive Critic.
