TLDR: ReMix is a novel off-policy reinforcement finetuning (RFT) method for Large Language Models (LLMs) that dramatically improves training efficiency and performance. By integrating a mix-policy proximal policy gradient with an increased Update-To-Data ratio, a KL-convex policy constraint, and policy reincarnation, ReMix effectively reuses historical data. This approach enables LLMs to achieve state-of-the-art reasoning abilities on math benchmarks with up to 450x less training data compared to existing on-policy methods, while also offering insights into how off-policy learning influences reasoning behaviors.
Large Language Models (LLMs) have shown incredible potential in complex reasoning tasks, but training them to achieve this capability, especially through Reinforcement Finetuning (RFT), often comes with a hefty price tag in terms of computational resources and time. This is largely due to the “on-policy” nature of most existing RFT methods, which means they don’t fully utilize data generated in past learning processes, leading to significant inefficiencies.
A new research paper titled “Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model” introduces a novel approach called Reincarnating Mix-policy Proximal Policy Gradient, or ReMix. This method aims to bring the benefits of “off-policy” reinforcement learning – learning from historical data – back into the spotlight for RFT, promising more economic and efficient scaling of LLMs.
Addressing the Efficiency Bottleneck
Traditional RFT methods, like PPO and GRPO, are on-policy, meaning they discard data after each training iteration. Imagine a sponge that you squeeze for water, but then throw away after one use, even if it’s still soaked. This is similar to how on-policy methods work, constantly needing fresh “squeezes” (new data generation) which is very expensive for large models and long reasoning processes.
ReMix tackles this by allowing these on-policy methods to effectively use “off-policy” data – data from previous learning steps. It’s like reusing the water from the sponge, making the process much more efficient. The paper highlights that while off-policy learning is known for its data efficiency, simply applying it can lead to instability or even a “collapse” in training. ReMix is designed to harness this efficiency while maintaining stability.
The Three Pillars of ReMix
ReMix achieves its goals through three main components:
1. Mix-policy Proximal Policy Gradient with Increased Update-To-Data (UTD) Ratio: This component is the core of ReMix. It strategically combines data generated by both the current and past policies for training. By increasing the UTD ratio, the model performs more gradient updates on existing data batches, significantly reducing the need for new, expensive data generation. This is where the “squeezing the soaked sponge” metaphor truly comes alive, maximizing the utility of every piece of data.
2. KL-Convex Policy Constraint: To prevent the model from deviating too far from its foundational knowledge (the base pre-trained model) or its most recent successful policies, ReMix introduces a dynamic constraint. This helps balance stability during training with the flexibility needed for the model to adapt and improve.
3. Policy Reincarnation: This unique feature allows for a seamless transition during training. Initially, the model benefits from the rapid improvements offered by the mix-policy approach. At a certain point, the “base model” is reset to the current, improved policy, and the training switches back to a more stable on-policy approach. This combines the best of both worlds: efficient early-stage learning and steady, long-term improvement.
Also Read:
- FR3E: Enhancing LLM Reasoning Through Structured Exploration
- Reinforcement Fine-tuning: A Robust Approach to Continual Learning in Large Language Models
Impressive Results and Efficiency Gains
The researchers conducted extensive experiments using ReMix on 1.5B and 7B base models, applying it to PPO and GRPO algorithms. They evaluated its performance on five challenging math reasoning benchmarks, including AIME’24, AMC’23, Minerva, OlympiadBench, and MATH500.
The results are striking. ReMix achieved state-of-the-art (SOTA) level performance, with an average Pass@1 accuracy of 52.10% for the 1.5B model and 63.27%/64.39% for the 7B model. More importantly, it demonstrated a massive reduction in training cost. For instance, ReMix-PPO achieved comparable performance to the strongest 1.5B competitor (DeepScaleR) with over a 30x reduction in rollout data volume. For 7B models, it showed an astounding 450x reduction compared to AceReason-Nemotron, while achieving slightly higher accuracy. Even against standard PPO, ReMix showed a 6x to 10x reduction in data volume.
Beyond just numbers, the research also provided insightful findings. They observed that off-policy learning can implicitly prefer shorter responses, a phenomenon they call the “Whipping Effect.” This can sometimes suppress the model’s self-reflection behavior, which is crucial for complex reasoning. However, ReMix manages to mitigate this, allowing the model to first become concise and then expand its responses for deeper reflection as needed.
The paper also notes that ReMix is more robust to variations in prompt templates, suggesting a more generalized understanding of reasoning rather than relying on explicit structural cues. This work represents a significant step forward in making advanced LLM training more accessible and sustainable.
For more technical details and a deeper dive into the methodology, you can find the full research paper here: Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model.
