TLDR: A new reinforcement learning algorithm, M2PO, enables stable and efficient training of large language models (LLMs) even when using significantly outdated data. It overcomes the ‘prosperity before collapse’ phenomenon observed with stale data by introducing a novel trust region mechanism that selectively masks only extreme outliers, preserving valuable learning signals and matching on-policy performance while drastically reducing token clipping.
Reinforcement Learning (RL) has become a cornerstone in advancing the reasoning capabilities of large language models (LLMs). However, a significant hurdle remains: most current RL algorithms for LLMs rely on what’s called ‘on-policy’ training. This means they constantly need fresh, newly generated data for every update, which can be incredibly inefficient and hard to scale, especially with the ever-growing size of LLMs.
To tackle this, researchers have explored ‘asynchronous RL systems’ that separate the process of generating data from the actual training. While this sounds promising for efficiency, its success depends on how well these algorithms can handle ‘stale data’ – information collected from older versions of the model. Unfortunately, existing methods often struggle here, either performing poorly or completely failing when faced with highly stale data.
The ‘Prosperity before Collapse’ Phenomenon
A recent research paper, “Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?”, delves into this challenge and uncovers a fascinating observation: ‘prosperity before collapse’. The authors, Haizhong Zheng, Jiawei Zhao, and Beidi Chen, found that stale data, if used correctly, can be just as informative as fresh, on-policy data. The catch is that without proper controls, training with stale data can initially show great promise (prosperity) but then quickly become unstable and fail (collapse).
The core issue with many existing algorithms, like Group Relative Policy Optimization (GRPO), lies in their ‘epsilon-clipping’ mechanism. This mechanism is designed to prevent overly large and unstable updates. However, when data is stale, this clipping disproportionately affects ‘high-entropy tokens’ – tokens that carry the most valuable and informative signals for model improvement. By clipping these crucial tokens, the algorithms inadvertently discard vital learning information, leading to degraded performance.
Introducing M2PO: A Novel Solution
Building on this insight, the researchers introduce a new algorithm called M2PO, which stands for Second-Moment Trust Policy Optimization. M2PO offers a more effective way to manage the ‘trust region’ in off-policy training, especially with stale data. Instead of relying on traditional batch-level KL divergence, which can suffer from cancellation effects and doesn’t adequately constrain tokens with large ratios, M2PO uses the ‘second moment of importance weights’ (M2) to measure the distribution gap between the old and new policies.
The M2 metric is particularly advantageous because it’s always non-negative and highly sensitive to outliers and noisy tokens. This allows M2PO to identify and suppress only the most extreme outliers while preserving the majority of informative updates. Essentially, M2PO employs a masking strategy that selectively excludes tokens until the batch-level M2 of the remaining tokens falls below a predefined threshold. This ensures stability without sacrificing valuable learning signals.
Also Read:
- Robust Preference Optimization: Enhancing LLM Alignment by Tackling Noisy Human Feedback
- ACPO: Enhancing Vision-Language Models for Complex Reasoning with Adaptive Learning
Remarkable Results and Practicality
The evaluation of M2PO across six different LLM scales (from 1.7 billion to 32 billion parameters) and eight mathematical reasoning benchmarks yielded impressive results. M2PO demonstrated stable off-policy training even with data stale by at least 256 model updates, consistently matching or even surpassing on-policy performance. For instance, it sharply reduced the fraction of clipped tokens from 1.22% to a mere 0.06% over training, effectively masking high-variance tokens while maintaining stable optimization.
Furthermore, M2PO proved to be robust and practical. Its sole threshold hyperparameter (Ï„M2) was found to be insensitive to variation, meaning a single setting worked effectively across all experiments. This highlights M2PO’s ease of use and reliability in diverse training scenarios. The paper’s findings suggest that M2PO is a significant step towards making off-policy RL a truly scalable solution for aligning and fine-tuning large language models, enabling efficient learning without the constant demand for fresh data. You can read the full research paper here: Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?
