TLDR: Adaptive Multi-Guidance Policy Optimization (AMPO) is a new framework for Large Language Models (LLMs) that improves reasoning by adaptively using guidance from multiple diverse teacher models. Unlike single-teacher methods, AMPO provides ‘guidance-on-demand’ only when the student model fails, and selects the most comprehensible reasoning paths. This approach significantly enhances performance on mathematical and out-of-distribution tasks, improves exploration, and is more data-efficient, offering a scalable way to achieve superior LLM reasoning and generalization.
Large Language Models (LLMs) have shown incredible potential in complex reasoning tasks, especially with techniques like Reinforcement Learning with Verifiable Rewards (RLVR). However, a common challenge is that these models often get stuck within their existing knowledge, struggling to explore new reasoning strategies or acquire novel information. Current methods frequently rely on a single ‘teacher’ model to guide them, which can limit the diversity of learning and introduce biases.
A new framework called Adaptive Multi-Guidance Policy Optimization (AMPO) aims to overcome these limitations. Drawing inspiration from how multiple teachers can enrich learning, AMPO introduces a novel approach that leverages guidance from several proficient teacher models, but only when the student model struggles to find correct solutions on its own. This ‘guidance-on-demand’ strategy encourages broader exploration while still valuing the student’s self-discovery.
How AMPO Works
AMPO’s core innovation lies in two main components. First, it uses an Adaptive Multi-Guidance Replacement mechanism. Imagine a student trying to solve a problem. If the student consistently fails to produce a correct answer, AMPO steps in. It replaces some of the student’s incorrect attempts with correct solutions from a diverse pool of teacher models. This intervention is strategic, only occurring when the student truly needs help, ensuring it learns from its mistakes without becoming overly reliant on external guidance.
Second, AMPO incorporates a Comprehension-based Guidance Selection mechanism. When multiple teacher solutions are available, how does the student choose which one to learn from? AMPO helps the student pick the reasoning path it is most likely to understand and assimilate. It does this by evaluating the student’s likelihood of generating the correct answer tokens given a teacher’s reasoning path. This ensures that the guidance provided is not just correct, but also comprehensible and effective for the student’s current learning stage.
Key Advantages and Results
Extensive experiments have shown that AMPO significantly improves LLM reasoning capabilities. For instance, it substantially outperforms a strong baseline (GRPO) with a 4.3% improvement on mathematical reasoning tasks and a remarkable 12.2% gain on out-of-distribution tasks. This means AMPO not only helps models solve familiar problems better but also enhances their ability to generalize to new, unseen challenges.
One of AMPO’s most impressive feats is its data efficiency. Using four peer-sized teacher models and a relatively small dataset of 8.5k samples, AMPO achieved performance comparable to approaches that use a single, more powerful teacher model (like DeepSeek-R1) with over five times more data (46k samples). This highlights a more efficient and scalable path to achieving superior reasoning and generalizability in LLMs.
Furthermore, AMPO boosts ‘Pass@k’ performance, indicating a greater capacity for generating diverse and correct solutions. It also maintains a higher ‘entropy’ during training, which means the model continues to explore a wider range of possibilities rather than collapsing into a narrow set of solutions. This balance between exploration and exploitation is crucial for robust, long-term performance gains.
AMPO also demonstrates efficiency in reasoning, generating shorter solutions compared to other methods while maintaining high performance. This translates to computational resource savings by producing more streamlined reasoning chains.
Also Read:
- RiskPO: Enhancing LLM Reasoning by Tackling Challenging Problems with Risk-Based Optimization
- DIVER: A New Approach to Enhance LLM Reasoning Through Diverse Exploration
Broader Impact
The research also explored the impact of different teacher compositions and the number of guidance instances provided. It consistently showed that a diverse set of teachers provides a more robust and effective learning signal than relying on a single, powerful expert. This principle has significant practical implications for developing more capable and generalizable LLMs.
AMPO represents a significant step forward in enhancing the reasoning ability of LLMs. By adaptively leveraging diverse guidance and focusing on comprehensible learning paths, it offers a powerful framework for models to transcend their inherent knowledge boundaries and achieve more robust, efficient, and generalizable reasoning. For more details, you can refer to the full research paper. Adaptive Multi-Guidance Policy Optimization.
