TLDR: MemoCoder is an AI-powered multi-agent system that automates code generation and repair. It uses specialized agents (Planner, Code Writer, Test Executor, Mentor) and a ‘Fixing Knowledge Set’ to learn from past errors and iteratively refine code. This approach significantly improves code correctness compared to traditional LLM methods, especially in complex scenarios requiring multiple attempts, by continuously adapting its fixing strategies.
In the rapidly evolving landscape of software development, Large Language Models (LLMs) like GitHub Copilot and ChatGPT have become indispensable tools for generating code. While these AI assistants excel at producing syntactically correct solutions for straightforward tasks, they often encounter difficulties with challenges that demand iterative debugging, robust error handling, or adaptation to diverse problem structures. Traditional methods, such as fine-tuning, are resource-intensive and lack flexibility, while self-repair strategies often fail to accumulate and reuse knowledge from previous attempts, leading to inefficiencies.
Addressing these limitations, researchers have introduced MemoCoder, an innovative multi-agent framework designed to automate function synthesis. MemoCoder stands out by enabling collaborative problem-solving among AI agents and facilitating persistent learning from past successful fixes. At its core is a ‘Fixing Knowledge Set,’ a memory module that stores successful repairs, making them retrievable for future tasks. A central ‘Mentor Agent’ plays a supervisory role, identifying recurring error patterns and refining high-level fixing strategies, guiding the self-repair process effectively.
MemoCoder’s architecture comprises four specialized LLM-based agents: the Planner, Code Writer, Test Executor, and Mentor. These agents work in concert to iteratively refine generated code until a correct solution is found or a predefined retry limit is reached.
How MemoCoder Works: A Collaborative Approach
The process begins with the Planner agent, which generates multiple algorithmic strategies for a given programming problem. This step is crucial as it helps anchor the LLM’s reasoning, reduces the likelihood of ‘hallucinations’ (incorrect or irrelevant code), and increases the chance of finding a viable solution from the outset.
Next, the Code Writer agent takes on two primary responsibilities: selecting one of the plans from the Planner and generating an initial code implementation. If this initial code fails to pass the guiding test case, the Code Writer enters a repair loop. In this loop, it refines the faulty code based on the original problem description, the initial code snippet, the guiding assertion, the specific error type, a detailed error message, and crucially, a ‘fixing suggestion’ provided by the Mentor agent.
The Test Executor agent is responsible for running the code produced by the Code Writer against a guiding assertion. If the code passes, it’s considered correct. If it fails, the Test Executor categorizes the failure into one of four types: Compile Error (syntax issues), Exceptions (runtime errors), Assertion Failure (incorrect results), or Timeout Errors (non-terminating code). This error information, along with the code, is then passed to the Mentor agent.
The Mentor Agent is the brain of MemoCoder’s learning mechanism. It analyzes new successful fixes, summarizes high-level causes of specific error types, and updates a short list of ‘fixing suggestions’ for those errors. Unlike the Code Writer, which focuses on individual errors, the Mentor identifies patterns across multiple repair attempts, refining and optimizing the system’s overall fixing strategies. This dynamic updating, combined with the ‘Fixing Knowledge Set,’ allows MemoCoder to continually learn and apply new and existing repair strategies without requiring costly retraining.
Evaluation and Performance
MemoCoder was rigorously evaluated across three public benchmarks: LiveCodeBench (LCB), Mostly Basic Python Problems (MBPP), and HumanEval, which cover a range of problem complexities. The system was compared against two strong baselines: ‘Zero-Shot’ prompting (where the LLM generates code without iterative refinement) and a ‘Self-Repair’ strategy (where the LLM iteratively refines its output without mentor-driven adaptation or external knowledge).
The results consistently show MemoCoder outperforming both baselines. For instance, improvements ranged from 3.1% to 12.1% in Pass@10 (at least one correct solution within 10 attempts) and from 1.4% to 14.5% in Pass@50 (at least one correct solution within 50 attempts). These gains highlight MemoCoder’s effectiveness in iteratively refining code and leveraging knowledge-guided code generation. While its Pass@1 (first attempt success) results were similar to Self-Repair, MemoCoder’s strength lies in its ability to converge on correct solutions through multiple, informed attempts.
The Impact of Each Component
An ablation study was conducted to understand the contribution of each major component. Removing the Planner significantly reduced the Pass@1 rate, underscoring its role in guiding accurate initial code generation. Conversely, removing the Retrieval-Augmented Generation (RAG) module or the Mentor’s Error Pattern Analysis did not significantly affect Pass@1 but critically impacted Pass@10 and Pass@50. This indicates that while planning helps with the first attempt, RAG and error pattern analysis are essential for the iterative fixing loops, significantly improving final solution correctness over multiple tries.
Also Read:
- Enhancing Code Review with Fine-Tuned Language Models: A C# Case Study
- Assessing LLM Code Generation: Introducing ReCatcher for Regression Testing
Understanding Error Evolution
The research also delved into how error types evolve during the code generation process. It was observed that over 50% of the initial code generations failed to compile, indicating basic syntax or structural errors. However, 17% of these ‘Not Compiled’ errors were resolved within the first two repair iterations. ‘Not Compiled’ and ‘Test Failed’ errors proved to be the most persistent, with over 70% of cases remaining in the same error type across iterations. In contrast, ‘Test Error’ (runtime exceptions) and ‘Timeout’ errors showed higher transition rates, often transforming into other error types rather than persisting, suggesting they are easier to modify but can introduce new issues.
MemoCoder represents a significant step forward in automated function synthesis, demonstrating the power of a multi-agent framework combined with persistent learning. By enabling AI agents to collaborate, learn from past mistakes, and adapt their strategies, MemoCoder offers a more robust and efficient approach to code generation and repair. For more details, you can refer to the full research paper: MemoCoder: Automated Function Synthesis using LLM-Supported Agents.
