TLDR: BAPO (Balanced Policy Optimization with Adaptive Clipping) is a novel method designed to stabilize off-policy reinforcement learning for large language models (LLMs). It addresses issues like imbalanced optimization (where negative samples dominate) and entropy collapse (reduced exploration) by dynamically adjusting clipping bounds. This approach re-balances positive and negative contributions, preserves policy entropy, and leads to faster, more stable, and data-efficient training. BAPO models have shown state-of-the-art performance on AIME benchmarks, outperforming both open-source and leading proprietary LLMs.
Reinforcement Learning (RL) has become a crucial technique for enhancing and aligning large language models (LLMs). However, applying RL in off-policy settings, where training uses older data from past policies, presents significant challenges. While off-policy RL improves how efficiently models learn from data, it often leads to instability, a sharp decline in the model’s ability to explore new options (policy entropy), and sometimes even a complete breakdown of the optimization process.
Researchers have identified two primary issues contributing to this instability. First, there’s an imbalance in the optimization process where samples with negative advantages (indicating poor actions) tend to dominate the policy gradient. This excessive penalization can suppress potentially useful behaviors and even cause gradients to explode, making training unstable. Second, a phenomenon called the Entropy-Clip Rule reveals that the fixed clipping mechanism used in common RL objectives, like PPO (Proximal Policy Optimization), systematically prevents updates that would increase policy entropy. This drives the model towards over-exploiting known good behaviors at the expense of exploring new, potentially better ones.
Introducing BAPO: Balanced Policy Optimization with Adaptive Clipping
To address these challenges, a new method called BAlanced Policy Optimization with Adaptive Clipping (BAPO) has been proposed. BAPO is a straightforward yet highly effective approach that dynamically adjusts the clipping bounds during training. This adaptive mechanism helps to re-balance the contributions of positive and negative samples, ensuring that useful behaviors are not suppressed and preventing gradient explosions. Crucially, BAPO also works to preserve policy entropy, maintaining a healthy balance between exploration and exploitation.
The core idea behind BAPO is to dynamically adjust the upper (c_high) and lower (c_low) clipping bounds based on the loss contributions from positive tokens. This allows the model to incorporate more low-probability positive tokens (which are often overlooked by fixed clipping) while filtering out excessive low-probability negative tokens. By doing so, BAPO maintains a smoother probability distribution, which is essential for preserving entropy and preventing the model from becoming too narrow in its responses.
Also Read:
- Bridging Language and Numbers: How New AI Training Boosts LLM Reasoning
- Scaf-GRPO: A Progressive Training Method for Advanced LLM Reasoning
Performance and Benefits
Extensive experiments across various off-policy scenarios, including sample replay and partial rollout, demonstrate BAPO’s effectiveness. When tested on benchmarks like AIME 2024 and AIME 2025, BAPO models consistently achieve fast, stable, and data-efficient training. For instance, a 7B BAPO model outperformed open-source counterparts like SkyWork-OR1-7B, and a 32B BAPO model not only achieved state-of-the-art results among models of its scale but also surpassed leading proprietary systems such as o3-mini and Gemini-2.5-Flash-Thinking.
BAPO offers several key advantages. It dynamically adjusts clipping bounds at each step, increasing the influence of positive tokens and preventing negative tokens from dominating the optimization. This adaptive approach also incorporates more low-probability positive tokens and filters out excessive low-probability negative tokens, both contributing to maintaining policy entropy. Furthermore, by setting a target contribution from positive tokens, BAPO prevents uncontrolled entropy growth and mitigates issues like tail degradation, where models perform well on easy problems but struggle with harder ones.
The method also proves robust in modern AI infrastructures that use techniques like partial rollout, where long trajectories are split into segments. This introduces off-policy learning, as different parts of a trajectory might come from outdated policies. BAPO shows greater stability in these settings compared to baseline algorithms. For more technical details, you can read the full research paper: BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping.
In summary, BAPO provides a significant step forward in stabilizing and improving off-policy reinforcement learning for LLMs. By addressing the fundamental issues of imbalanced optimization and entropy collapse through adaptive clipping, it enables more robust, efficient, and high-performing language models.
