TLDR: CausalPlan is a new framework that improves the collaboration of Large Language Model (LLM) agents by integrating explicit structural causal reasoning into their planning process. It addresses the common issue of LLMs generating causally invalid actions by learning a causal graph from agent trajectories. This graph helps guide action selection by assigning causal scores to LLM-generated proposals, reweighting them, or providing causally grounded alternatives when needed. Evaluated on the Overcooked-AI benchmark with various open-source LLMs, CausalPlan consistently reduced invalid actions and improved collaboration in both AI-AI and human-AI settings, outperforming traditional reinforcement learning baselines.
Large Language Models, or LLMs, are increasingly being used as decision-makers in complex tasks, especially when multiple AI agents need to work together. While these models show great promise, particularly in collaborative environments, they often struggle with a fundamental issue: generating actions that don’t make causal sense. This means an LLM agent might try to perform an action before its necessary prerequisites are met, like trying to put an onion in a pot before picking it up. This limitation is particularly noticeable in smaller, open-source LLMs, hindering their ability to coordinate effectively and plan in dynamic situations.
To tackle this challenge, researchers from Deakin University’s Applied Artificial Intelligence Initiative (A2I2) — Minh Hoang Nguyen, Van Dai Do, Dung Nguyen, Thin Nguyen, and Hung Le — have introduced a new framework called CausalPlan. This innovative two-phase system aims to embed explicit structural causal reasoning directly into the LLM’s planning process, without needing to fine-tune the LLM itself. You can read their full paper, CausalPlan: Empowering Efficient LLM Multi-Agent Collaboration Through Causality-Driven Planning, for all the technical details.
How CausalPlan Works
At the heart of CausalPlan is the Structural Causal Action (SCA) model. This model learns a ‘causal graph’ by observing how agents behave over time. Think of a causal graph as a map that shows how previous actions and the current state of the environment influence future decisions. For example, it might learn that picking up an onion strongly influences the ability to put that onion into a pot, but has no influence on filling a dish with soup.
CausalPlan operates in two main phases:
1. Causal Action Structure Learning: In this initial phase, the SCA model is trained. It analyzes past agent behaviors and environmental states to build its causal understanding. This results in a ‘Causal Action Matrix’ (M), which essentially assigns a ‘causal score’ to every possible next action based on the current situation and what happened previously. This matrix tells the system how likely an action is to be causally valid and effective.
2. Agent Planning with Causal Knowledge: This is where the magic happens during real-time decision-making. When an LLM agent proposes a set of actions, CausalPlan steps in. It uses the causal scores from the matrix M to adjust the LLM’s proposed action probabilities. If the LLM suggests actions that are causally sound, their probabilities are reweighted to favor them. Crucially, if the LLM ‘hallucinates’ or fails to propose any valid actions, CausalPlan has a ‘Causal Backup Action’ mechanism. In such cases, it falls back to selecting the action with the highest causal probability from its learned knowledge, ensuring the agent always makes a causally grounded move.
Also Read:
- Enhancing LLMs: How Causal Reasoning Reduces Hallucinations
- Bridging the Language Gap: How AI is Enabling Robots to Understand Human Instructions
Real-World Evaluation and Impact
The researchers put CausalPlan to the test on the Overcooked-AI benchmark, a popular environment for multi-agent coordination tasks. They evaluated it across five different tasks and four open-source LLMs of varying sizes: Gemma-7B, Llama-8B, Qwen-14B, and Llama-70B. The experiments covered both AI-AI and human-AI collaboration scenarios.
The results were highly encouraging. CausalPlan consistently reduced the number of invalid actions and significantly improved collaboration across all tested LLMs and tasks. It even outperformed strong reinforcement learning baselines, which are traditional methods for training agents in such environments. For instance, in the Asymmetric Advantages layout, CausalPlan showed a 63% advantage over the next best baseline. Even for smaller LLMs, CausalPlan provided notable benefits without requiring any fine-tuning of the models themselves.
This work highlights the immense value of integrating causality-driven planning into LLM-based multi-agent systems. By enabling LLMs to reason about cause and effect, CausalPlan paves the way for more efficient, interpretable, and generalizable collaborative AI. While still an initial step, this framework holds significant potential for improving the safety and effectiveness of future AI systems that need to work together, whether with other AIs or with humans.
