TLDR: EAPO is a new reinforcement learning framework that improves large language models’ reasoning abilities by allowing them to consult external experts during training. This adaptive, on-demand assistance provides richer feedback, making exploration more efficient. Over time, the model internalizes this expert knowledge, enabling it to solve complex problems independently and more accurately during evaluation, outperforming existing methods on mathematical reasoning tasks.
Large language models (LLMs) have shown impressive capabilities in complex tasks like mathematical and logical reasoning. However, training these models to produce accurate, multi-step reasoning chains using traditional reinforcement learning (RL) methods often faces significant hurdles. These challenges include navigating vast search spaces, dealing with infrequent or “sparse” feedback (rewards), and maintaining stable optimization during training.
Existing approaches typically fine-tune models with supervised learning and then refine them using reinforcement learning from human feedback (RLHF). While effective, RLHF can suffer from issues like “reward hacking” (where the model finds loopholes to get rewards without truly solving the problem), over-optimizing for proxy objectives, and instability during the training process. For reasoning models, these problems are particularly acute, leading to high costs for exploration and unpredictable behavior.
To tackle these limitations, researchers have introduced a new framework called Expert-Assisted Policy Optimization (EAPO). This innovative RL framework enhances how models explore and learn by allowing them to interact with external “experts” during training. Unlike previous methods where models reason in isolation, EAPO encourages the policy to adaptively decide when and how to consult these experts. This interaction provides richer reward signals and helps the model develop more reliable reasoning paths.
The core idea behind EAPO is to treat “consulting experts” as a learnable action within the model’s decision-making process. During training, if the model encounters a difficult problem or an ambiguous reasoning path, it can choose to ask for a hint from an expert. This external assistance helps guide the reasoning process, making exploration more efficient and stable. Crucially, EAPO is designed so that the expert knowledge is eventually internalized into the policy model. This means that during evaluation, the model can solve questions independently, without needing any external help, yet still produce improved reasoning and more accurate solutions.
EAPO employs a curriculum-like optimization design. In the early stages of training, the model is allowed to consult experts freely, receiving abundant assistance. As training progresses, a penalty term for consultation is introduced, and the model is incentivized to consult less frequently. This gradual reduction in reliance encourages the model to internalize the expert strategies. Experimental observations show that the policy model learns to consult experts only for challenging questions, rather than for all cases, demonstrating its growing self-sufficiency.
The framework supports multi-expert parallel reasoning, where the policy model can query multiple experts simultaneously to gather broader information. This parallel evidence gathering improves the coverage of reasoning threads and reduces reliance on any single path. The effectiveness of EAPO is also influenced by the size and capacity of these expert models; larger and more capable experts provide higher quality guidance, leading to better performance.
Experiments were conducted on challenging mathematical reasoning benchmarks, including AIME 2024, AIME 2025, and AIMO 2025. EAPO consistently outperformed various strong baselines, such as expert-assisted workflows, expert-distilled models, and self-exploratory RL models. On average, EAPO showed a significant gain of 5 points over self-exploratory models. Furthermore, models trained with EAPO exhibited lower variance on test sets, indicating greater stability and consistency in their reasoning performance.
Also Read:
- CLPO: A Self-Evolving Learning Approach for Enhanced LLM Reasoning
- Enhancing LLM Reasoning with Adaptive Advantage Estimation
In essence, EAPO offers a powerful way to enhance the reasoning capabilities of large language models. By strategically integrating on-demand expert assistance during training and then gradually internalizing that knowledge, EAPO enables models to achieve superior accuracy and stability, ultimately leading to more robust and autonomous reasoning agents. For more details, you can refer to the full research paper here.
