TLDR: Single-stream Policy Optimization (SPO) is a new method for training Large Language Models (LLMs) that addresses inefficiencies in existing group-based approaches like GRPO. SPO uses a persistent, adaptive value tracker and global advantage normalization, along with prioritized sampling, to provide a stable and efficient learning signal. This group-free design eliminates computational waste from degenerate groups and synchronization bottlenecks, leading to significantly higher training throughput and improved performance on complex reasoning tasks compared to GRPO.
In the rapidly evolving field of artificial intelligence, Large Language Models (LLMs) are at the forefront, demonstrating remarkable capabilities in reasoning and problem-solving. A key driver behind these advancements is Reinforcement Learning (RL), a training paradigm that helps LLMs learn from verifiable rewards. However, current RL methods, particularly those relying on ‘group-based’ approaches like Group Relative Policy Optimization (GRPO), face significant hurdles that limit their efficiency and scalability.
Researchers have identified two primary inefficiencies in these prevailing methods. Firstly, group-based approaches often suffer from ‘degenerate groups.’ This occurs when all responses generated by an LLM within a group for a given prompt yield the same outcome – for instance, all correct or all incorrect. In such scenarios, the learning signal collapses to zero, effectively wasting valuable computation and data. Secondly, these methods introduce ‘synchronization barriers’ in distributed training environments. Imagine a team working on a project where everyone has to wait for the slowest member to finish before the next step can begin. This is precisely what happens in group-based LLM training, especially in complex tasks involving multiple steps or tool use, where generation times can vary wildly. A single slow response can stall an entire group, severely impacting training speed and scalability.
Introducing Single-stream Policy Optimization (SPO)
To overcome these fundamental limitations, a new approach called Single-stream Policy Optimization (SPO) has been proposed. This method marks a deliberate return to the classic ‘single-stream’ paradigm, where each training sample is treated as an independent prompt-response pair. SPO is designed to eliminate the issues of degenerate groups and synchronization barriers, offering a more robust and efficient path for LLM reasoning.
SPO achieves its stability and efficiency through three core components. First, it employs a lightweight, KL-adaptive value tracker. Instead of relying on noisy, on-the-fly baselines from small groups, this tracker maintains a persistent, historically informed estimate of the success probability for each prompt. This provides a stable, low-variance baseline that adapts dynamically as the LLM’s policy evolves. Second, SPO implements global advantage normalization. Rather than normalizing advantages within small, unstable groups, it normalizes them across an entire batch of prompts. This ensures a more consistent and stable learning signal for every sample. Finally, SPO incorporates prioritized prompt sampling, an adaptive curriculum strategy. This mechanism intelligently focuses computational resources on prompts that offer the highest learning potential, prioritizing those with high uncertainty while ensuring a minimum level of exploration. This means the model spends less time on tasks it has already mastered or those that are currently too difficult to yield useful signals.
Tangible Advantages and Performance Gains
The benefits of SPO’s principled design are clear and impactful. Its ‘group-free’ architecture inherently eliminates the synchronization bottlenecks that plague group-based methods. In practical simulations modeling agentic training scenarios with variable interaction times, SPO demonstrated a significant 4.35 times speedup in training throughput. This is because it can process samples asynchronously, collecting the fastest ones without waiting for stragglers, which is crucial for complex tasks like multi-turn tool use or long-horizon reasoning.
Furthermore, SPO significantly improves signal efficiency and stability. While GRPO often sees 60-80% of its samples fall into degenerate groups, yielding no learning signal, SPO’s near-zero advantages indicate accurate predictions rather than wasted computation. The history-informed baseline in SPO provides a stable, low-variance learning signal, reducing advantage variance by nearly 50% compared to raw rewards. In contrast, GRPO’s on-the-fly baseline often produces a highly volatile signal, even when it does generate one.
Empirical evaluations using the Qwen3-8B LLM on challenging math competition benchmarks (AIME 24, AIME 25, BeyondAIME, BRUMO 25, and HMMT 25) confirm SPO’s superiority. SPO consistently outperformed GRPO, achieving an average improvement of +3.4 percentage points on the maj@32 metric. Notably, it showed substantial gains on BRUMO 25 (+7.3 pp), AIME 25 (+4.4 pp), and HMMT 25 (+3.3 pp). The pass@k curves also demonstrated a consistent advantage for SPO across all evaluated k values, with an average improvement of approximately 2.4 percentage points.
Also Read:
- Collaborative AI Agents Enhance Prompt Optimization for Large Language Models
- Enhancing LLM Reasoning with Latent Thought Optimization
A Step Towards More Efficient LLM Training
The success of Single-stream Policy Optimization challenges the prevailing trend of adding incidental complexity to RL algorithms for LLMs. Instead, it highlights that a return to fundamental reinforcement learning principles, coupled with thoughtful design, can lead to more robust, scalable, and efficient training methods. This work provides a strong foundation for future research in agentic and reasoning model training, paving the way for the next generation of LLMs. You can read the full research paper for more details here: Single-stream Policy Optimization.
