TLDR: A new research paper introduces the Online Agent (OA), a novel approach to Continual Reinforcement Learning (CRL) that prevents AI agents from forgetting previously learned tasks. OA achieves this by using an online, shallow “Follow-The-Leader” world model for planning, which is inherently immune to catastrophic forgetting. The paper also presents “Continual Bench,” a new benchmark environment for evaluating CRL. Empirical results show OA’s superior performance in learning new tasks while retaining old skills, outperforming other continual learning methods.
In the rapidly evolving field of artificial intelligence, a significant challenge for learning agents is the ability to continuously acquire new skills without forgetting previously mastered ones. This problem, known as catastrophic forgetting, is a major hurdle in Continual Reinforcement Learning (CRL), where an AI agent is expected to endlessly adapt and solve multiple tasks presented sequentially.
A new research paper, titled “Continual Reinforcement Learning by Planning with Online World Models,” proposes an innovative solution to this challenge: the Online Agent (OA). Developed by Zichen Liu, Guoji Fu, Chao Du, Wee Sun Lee, and Min Lin, this approach tackles forgetting by enabling AI agents to plan using online world models.
The Online Agent’s Core Innovation
The Online Agent’s key lies in its ability to learn a “Follow-The-Leader” (FTL) shallow model online. This model is designed to capture the dynamics of the world, allowing the agent to plan its actions using a technique called Model Predictive Control (MPC). A crucial advantage of this online world model is its inherent immunity to forgetting, a property that is theoretically supported by a proven regret bound, ensuring the model continuously improves without losing past knowledge.
Unlike traditional methods that might train separate components for each task or rely on extensive memory buffers, the OA’s planner makes decisions solely based on the most up-to-date online model. This incremental update mechanism allows the agent to evolve seamlessly, integrating new information from real-world interactions to refine its understanding of the environment, which directly benefits its planning for subsequent actions.
Introducing Continual Bench
To rigorously evaluate the Online Agent’s capabilities in CRL settings, the researchers also developed a dedicated environment called “Continual Bench.” This benchmark addresses limitations found in previous evaluation platforms by focusing on a unified world dynamics, which is essential for studying both forgetting and knowledge transfer in a realistic yet computationally lightweight manner.
Continual Bench features six diverse tasks, including ‘pick-place,’ ‘button-press,’ ‘door-open,’ ‘peg-unplug,’ ‘window-close,’ and ‘faucet-close.’ These tasks are spatially arranged to maximize the distance between adjacent tasks, thereby increasing the non-stationarity and providing a robust test for an agent’s ability to retain skills across significant shifts in its operational environment. The design ensures a consistent state space, avoiding the physical conflicts that could hinder meaningful transfer or forgetting studies in other benchmarks.
Also Read:
- Unlocking Real-World Potential: Adaptability in Multi-Agent Reinforcement Learning
- Accelerating Offline Reinforcement Learning with Single-Step Trajectory Planning
Empirical Success and Future Promise
In experiments conducted on Continual Bench, the Online Agent demonstrated superior performance compared to several strong baselines, including methods based on regularization, replay, and architectural modifications, all operating within the same model-planning framework. While other agents struggled with catastrophic forgetting, showing diminished performance on old tasks as new ones were introduced, OA consistently maintained high performance across all tasks it had encountered. Remarkably, OA achieved performance comparable to a “Perfect Memory” baseline, which retains all past data, but did so with significantly more efficient online updates and constant computational overhead.
This research marks a significant step towards building truly autonomous AI agents that can continuously learn and adapt throughout their lifetime without succumbing to the common problem of forgetting. The Online Agent’s robust performance and the introduction of Continual Bench provide valuable contributions to the ongoing development of continual reinforcement learning. For more details, you can read the full research paper here.
