TLDR: Multi-Agent Evolve (MAE) is a new framework enabling Large Language Models (LLMs) to self-improve their reasoning capabilities across diverse tasks like math, reasoning, and general knowledge. It uses a triplet of interacting LLM agents (Proposer, Solver, Judge) that co-evolve without human-curated data or external verifiers. The Proposer generates questions, the Solver attempts solutions, and the Judge evaluates both, providing self-rewarding signals. Experiments show MAE improves LLM performance significantly, outperforming supervised fine-tuning and other baselines, demonstrating a scalable and data-efficient approach to enhancing general LLM intelligence.
A new research paper introduces an innovative framework called Multi-Agent Evolve (MAE) that allows large language models (LLMs) to improve their reasoning capabilities without relying on extensive human-curated datasets or verifiable rewards. This approach addresses a significant challenge in the field of artificial intelligence, where the scalability and generality of LLM training have often been limited by the need for meticulously labeled data.
Traditional reinforcement learning (RL) methods for LLMs, while effective in tasks like coding, typically require ground truth answers to provide clear reward signals. Similarly, recent Self-Play RL techniques, inspired by successes in games like Go, often depend on grounded environments such as Python interpreters or game engines for feedback. Extending these methods to more general, open-ended domains like natural language reasoning has proven difficult due to the ambiguous nature of real-world reward signals.
MAE tackles this by instantiating a triplet of interacting agents—a Proposer, a Solver, and a Judge—all derived from a single base LLM. These agents engage in a continuous cycle of self-improvement. The Proposer’s role is to generate new, challenging questions. The Solver then attempts to provide solutions to these questions. Finally, the Judge evaluates both the generated questions and the proposed solutions, providing crucial feedback that drives the learning process.
The core of MAE’s design lies in its self-rewarding mechanisms, which are domain-agnostic. The Judge, operating under the ‘LLM-as-a-Judge’ paradigm, assigns scores based on the quality and correctness of answers and questions, eliminating the need for human labels or external verifiers. The Proposer is incentivized not only for generating high-quality questions but also for creating questions that are difficult for the current Solver, fostering an adversarial co-evolution. This dynamic pushes the Solver to improve its abilities to tackle more complex problems, while the Proposer learns to craft increasingly sophisticated challenges. Format rewards are also integrated to ensure that the agents’ outputs are structured correctly, maintaining the stability of the training loop.
Experiments conducted on the Qwen2.5-3B-Instruct model demonstrated the effectiveness of MAE. Even starting with a minimal setup and no real-world data, MAE showed an average improvement of 4.54% across various benchmarks, including mathematics, reasoning, and general knowledge Q&A. When initialized with a small set of unlabeled reference questions, its performance was further enhanced. Notably, MAE variants consistently outperformed standard Supervised Fine-Tuning (SFT) on the same dataset, despite SFT relying on ground-truth answers while MAE does not.
The research highlights MAE as a scalable and data-efficient method for enhancing the general reasoning abilities of LLMs with minimal human supervision. The framework’s stability is maintained through mechanisms like quality filtering, which ensures only high-quality questions enter the training dataset, preventing degradation over prolonged training periods. The paper also provides a detailed analysis of the training process, showing how the Proposer learns to generate questions of desirable difficulty, which in turn benefits the Solver’s learning curve.
Also Read:
- Enhancing Mathematical Reasoning in Language Models: A Reinforcement Learning Approach to Budget Forcing
- Co-Sight: A Framework for Trustworthy and Efficient AI Agent Reasoning
This work represents a significant step towards building LLMs that can continuously self-improve in diverse and open-ended domains, moving beyond the limitations of human-annotated data and external verifiers. For more in-depth information, you can read the full research paper here.
