TLDR: ReasoningBank is a new memory framework for LLM agents that distills generalizable reasoning strategies from both successful and failed experiences. It allows agents to continuously learn and evolve by retrieving relevant memories for new tasks and integrating new learnings back into the system. Coupled with Memory-aware Test-Time Scaling (MaTTS), which generates diverse experiences for better memory curation, ReasoningBank significantly improves agent effectiveness and efficiency across various benchmarks, enabling emergent, adaptive behaviors.
Large language model (LLM) agents are becoming increasingly common in real-world applications, handling a continuous flow of tasks. However, a significant challenge they face is their inability to learn from past interactions. This means they often repeat mistakes and discard valuable insights, hindering their ability to improve over time.
To address this, researchers have introduced ReasoningBank, a new memory framework designed to help agents learn and evolve. ReasoningBank works by extracting generalizable reasoning strategies from an agent’s experiences, both successful and failed. Instead of just storing raw interaction histories or only successful routines, ReasoningBank distills higher-level, transferable patterns.
Here’s how it works: when an agent encounters a new task, it retrieves relevant memories from ReasoningBank to guide its actions. After completing the task, the new experience is analyzed, and new learnings are distilled and integrated back into ReasoningBank. This creates a continuous learning loop, allowing the agent to become more capable over time. A key aspect is its ability to learn from failures, turning past mistakes into preventative lessons, which is a significant improvement over previous memory systems that often overlooked these valuable insights.
Building on ReasoningBank, the researchers also developed Memory-aware Test-Time Scaling (MaTTS). This approach accelerates and diversifies the learning process by scaling up the agent’s interaction experience. By allocating more computational resources to each task, the agent can generate a wider range of diverse experiences. These experiences provide rich “contrastive signals” – essentially, comparisons between different outcomes – which help synthesize higher-quality memory. In turn, this improved memory guides more effective scaling, creating a powerful synergy between memory and test-time scaling.
The study highlights two main ways MaTTS scales experience: parallel scaling and sequential scaling. Parallel scaling involves generating multiple trajectories (attempts) for the same task simultaneously. By comparing these different attempts, the agent can identify consistent successful patterns and filter out less effective solutions. Sequential scaling, on the other hand, involves iteratively refining the agent’s reasoning within a single trajectory after its initial completion. This process uses intermediate notes and corrections as valuable signals for memory, capturing insights that might not appear in the final solution.
Experiments were conducted on challenging benchmarks, including web browsing tasks (WebArena, Mind2Web) and software engineering tasks (SWE-Bench-Verified). ReasoningBank consistently outperformed existing memory mechanisms, showing improvements in both effectiveness (up to 34.2% relative improvement) and efficiency (16.0% fewer interaction steps). MaTTS further amplified these gains, demonstrating that memory-driven experience scaling is a new and effective dimension for agent improvement.
Also Read:
- UniAPL: Unifying Language Model Training for Enhanced Instruction Following
- Why Advanced AI Models Struggle with Simple Visual Tasks: The Serial Processing Gap
Emergent Behaviors and Learning from Failure
A fascinating aspect of ReasoningBank is how it enables emergent behaviors. The strategies stored in ReasoningBank are not static; they evolve over time. Initially, they might be execution-oriented, like “find navigation links.” With more experience, they progress to adaptive self-reflections, such as “re-verify identifiers.” Eventually, they mature into complex, compositional strategies like “cross-referencing task requirements and reassessing options.” This evolution shows how agents can refine their strategies from basic actions to high-level reasoning.
The research also emphasizes the importance of learning from failures. Unlike other methods that only focus on successful trajectories, ReasoningBank actively distills lessons from failed attempts. This allows the system to transform failures into constructive signals, leading to more robust generalization. Furthermore, the efficiency study revealed that ReasoningBank significantly reduces the number of steps required for successful task completion, indicating that agents are guided to more effective reasoning paths rather than just cutting short failed attempts.
In conclusion, ReasoningBank and MaTTS offer a promising pathway toward building adaptive and lifelong-learning agents. By distilling strategy-level reasoning from both successes and failures and integrating it with test-time scaling, agents can continuously evolve, improve performance, and reduce redundant exploration. You can read the full research paper for more technical details here: ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory.
