TLDR: MUSE is a new AI agent framework that learns from experience to tackle complex, long-horizon tasks. It uses a hierarchical Memory Module (Strategic, Procedural, Tool Memory) and a “Plan-Execute-Reflect-Memorize” loop to continuously improve. Evaluated on the TAC benchmark, MUSE achieved state-of-the-art performance, demonstrating strong continuous learning and generalization capabilities, even with a lightweight LLM.
Large Language Models (LLMs) have shown incredible abilities across many areas, but they still face big challenges when used as AI agents for complex, real-world tasks that take a long time to complete. A major problem with current LLM agents is that they are static; they can’t learn from their experiences or continuously improve as they work.
To tackle this, researchers have introduced a new framework called MUSE, which stands for Memory-Utilizing and Self-Evolving. MUSE is designed to be an experience-driven, self-evolving system that can learn and adapt over time, much like humans do. Its core is a hierarchical Memory Module that organizes different levels of experience.
How MUSE Works
MUSE operates through a continuous “Plan-Execute-Reflect-Memorize” loop. When given a new task, the Planning-Execution (PE) Agent first breaks it down into smaller sub-tasks. For each sub-task, the PE Agent uses its accumulated knowledge from the Memory Module to plan and carry out actions in an interactive environment, which can involve various software like chat applications, code editors, and web browsers.
After each sub-task is attempted, a Reflect Agent steps in. This agent autonomously assesses the execution, checking if the sub-task was successful. If it was, the Reflect Agent converts the successful actions into structured experience and integrates it back into the Memory Module. If the sub-task failed, the Reflect Agent diagnoses the failure and instructs the PE Agent to replan and retry. This continuous feedback loop allows MUSE to learn from both successes and failures, constantly refining its approach.
The Memory Module
The Memory Module is central to MUSE’s ability to learn on the job without needing to fine-tune the underlying LLM. It’s composed of three distinct types of memory:
- Strategic Memory: This stores high-level guidance and solutions for dilemmas encountered during tasks, especially those requiring multiple attempts. It captures “problem-solution” patterns.
- Procedural Memory: This acts as a knowledge base of Standard Operating Procedures (SOPs), archiving successful sub-task trajectories. It’s indexed by application and provides detailed operational steps.
- Tool Memory: This is like the agent’s “muscle memory” for using individual tools. It includes static descriptions of tools and dynamic instructions that guide the agent’s next immediate action after a tool is used.
All these memory types are maintained in natural language, making the accumulated knowledge LLM-agnostic, meaning experience gained by one model can be used by another.
Impressive Results
MUSE was evaluated on TheAgentCompany (TAC) benchmark, a challenging set of 175 long-horizon productivity tasks that simulate a corporate environment. The results were quite remarkable:
- Continuous Learning: Experiments showed that MUSE’s performance steadily improved across iterations as it accumulated experience, outperforming a memory-less baseline by over 10%. This demonstrates its self-evolving capability.
- Generalization: The knowledge acquired by MUSE proved to be highly generalizable. When tested on a set of difficult, unseen tasks, the agent with pre-learned memory significantly outperformed the same agent without memory, achieving an impressive 33.41% partial completion score compared to 23.65% without memory.
- State-of-the-Art Performance: On the full TAC benchmark, MUSE achieved a new state-of-the-art performance with an average partial completion score of 51.78%. This is the first time an agent has surpassed the 50% threshold on this benchmark, outperforming previous top models by nearly 20%, even using a lighter Gemini-2.5 Flash model.
Ablation studies further confirmed the importance of the Reflect Agent and demonstrated that the MUSE architecture is adaptable to different LLMs, including open-source models like DeepSeek-V3, showing significant performance boosts with the memory module.
Also Read:
- ProSEA: A New AI Framework for Adaptive Problem Solving Through Exploration
- Structured Cognitive Loop: A New Blueprint for Reliable LLM Agents
Looking Ahead
MUSE represents a significant step towards creating AI agents that can truly “learn on the job” and continuously improve their capabilities in real-world scenarios. While the current architecture is fully autonomous, its design also allows for future integration of human feedback, paving the way for collaborative human-agent systems. This research establishes a new way forward for AI agents capable of automating complex productivity tasks. You can read the full research paper for more details. Read the full paper here.
