TLDR: Asymmetric Proximal Policy Optimization (AsyPPO) is a new framework that improves Large Language Model (LLM) reasoning by reintroducing efficient “critic” models. Instead of a single large critic, AsyPPO uses multiple lightweight “mini-critics” trained on disjoint data subsets to provide diverse and accurate value estimations. It also refines policy updates by using inter-critic agreement to mask low-informative states and divergence to filter out noisy states from exploration. This approach significantly boosts performance and stability on various benchmarks while being computationally more efficient than traditional PPO.
The field of Reinforcement Learning for Large Language Models (RL4LLM) has seen significant advancements, yet it faces a persistent challenge: the computational cost and inefficiency of traditional “critic” models. These critics are crucial for evaluating actions and guiding the learning process in PPO (Proximal Policy Optimization), a powerful algorithm. Historically, in LLM applications, these critics have often been sidelined due to their large size and the difficulty in training them effectively with sparse rewards and long reasoning chains.
A new framework, Asymmetric Proximal Policy Optimization (AsyPPO), reintroduces the critic’s vital role in a more efficient and scalable manner. Developed by researchers from Hong Kong University of Science and Technology, Mila, Université de Montréal, and Alibaba Group, AsyPPO addresses the computational bottleneck by employing a set of lightweight “mini-critics” instead of a single, large one.
The core idea behind AsyPPO is to leverage the inherent representational ability of pre-trained LLMs. Unlike traditional deep RL where agents learn from scratch, LLMs already possess a rich understanding of language. This allows smaller critics to provide meaningful guidance to much larger “actor” models (the LLMs generating text). However, a single small critic can still struggle with accuracy, especially in complex reasoning tasks.
To overcome this, AsyPPO uses an ensemble of mini-critics. A key innovation is the “group-level non-overlapping data division” technique. Instead of training all critics on the same data, each mini-critic is trained on a distinct subset of responses from each prompt. This encourages diversity among the critics, preventing them from behaving identically and leading to more robust and reliable value estimations. The research found that using two mini-critics strikes an optimal balance between correction capability and computational efficiency. For instance, two Qwen3-1.7b-Base critics could effectively guide a larger Qwen3-14b-Base policy, outperforming traditional symmetric PPO. This asymmetric design significantly reduces peak memory usage by 20% and accelerates training by about 20 seconds per step.
Beyond robust value estimation, AsyPPO further refines the policy learning process by utilizing the agreement and divergence among these mini-critics. When critics strongly agree on a state’s value, it often indicates a low-informative state where further learning signals are minimal. AsyPPO masks advantages in these states, preventing the policy from overfitting to redundant samples and improving training stability. Conversely, when critics diverge significantly, it suggests uncertainty or that the state might be reasoning-independent or noisy. AsyPPO filters out such high-divergence states from entropy regularization, which helps in suppressing spurious exploration and promotes safer, more effective learning.
The effectiveness of AsyPPO has been demonstrated across multiple benchmarks, including MATH-500, OlympiadBench, MinervaMath, and AMC 2023. After training on just 5,000 samples from open-source data, AsyPPO consistently improved learning stability and performance. For example, it achieved performance gains of over 6% on Qwen3-4b-Base and about 3% on Qwen3-8b-Base and Qwen3-14b-Base compared to classic PPO, without needing additional complex tricks. This highlights the importance of architectural innovations for developing scalable and efficient algorithms in RL4LLM.
Also Read:
- Boosting LLM Reasoning with Adaptive Multi-Teacher Guidance
- Optimizing Diffusion LLMs for Rapid and Robust Reasoning
AsyPPO represents a significant step forward in making reinforcement learning more practical and effective for large language models. By intelligently redesigning the critic architecture and leveraging inter-critic uncertainty, it offers a path to more stable, efficient, and powerful LLM reasoning. You can read the full paper here: Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning.
