TLDR: Group Sequence Policy Optimization (GSPO) is a new reinforcement learning algorithm for training large language models. Unlike previous methods that use token-level importance ratios, GSPO defines importance ratios based on sequence likelihood and performs sequence-level clipping and optimization. This approach significantly improves training stability, efficiency, and performance, especially for Mixture-of-Experts (MoE) models, by resolving issues like model collapse and expert-activation volatility. GSPO has contributed to the improvements in the latest Qwen3 models and simplifies RL infrastructure.
In the rapidly evolving field of artificial intelligence, reinforcement learning (RL) has become a cornerstone for enhancing the capabilities of large language models (LLMs). These models are now tackling complex problems like advanced mathematics and programming, requiring deeper and longer reasoning processes. However, scaling RL training for these gigantic models has faced a significant hurdle: instability.
A recent research paper, Group Sequence Policy Optimization, introduces a groundbreaking algorithm called Group Sequence Policy Optimization (GSPO) that promises to make RL training for LLMs more stable, efficient, and performant. This work, primarily from the Qwen Team at Alibaba Inc., addresses critical limitations found in previous state-of-the-art RL algorithms, such as Group Relative Policy Optimization (GRPO).
The Challenge with Existing RL Algorithms
Current RL methods, exemplified by GRPO, often suffer from severe stability issues when applied to very large language models. This can lead to what researchers call “catastrophic and irreversible model collapse,” hindering further improvements. The core problem identified by the researchers is a fundamental misapplication of importance sampling weights. In GRPO, these weights are applied at the individual token level within a sequence. Because each token’s weight is based on a single sample, it introduces high-variance noise into the training process. This noise accumulates over long responses and is made worse by the clipping mechanism, ultimately causing the model to fail.
This instability is particularly pronounced in Mixture-of-Experts (MoE) models, which have a sparse activation nature. The experts activated for a given response can change significantly after just one or two gradient updates, causing token-level importance ratios to fluctuate wildly and further invalidating them. To counteract this, GRPO previously required complex strategies like “Routing Replay,” which involved caching and reusing activated experts, adding memory and communication overhead and potentially limiting the model’s true capacity.
Introducing Group Sequence Policy Optimization (GSPO)
GSPO offers a fresh perspective by aligning the unit of optimization with the unit of reward. Instead of focusing on individual tokens, GSPO defines the importance ratio based on the *entire sequence likelihood*. This means it considers how much a generated response, sampled from an older version of the model, deviates from the current model’s expectations for that whole sequence.
The algorithm then performs clipping, rewarding, and optimization at the sequence level. This approach naturally aligns with the principle of importance sampling, where the reward is granted to the entire sequence. By weighting all tokens in a response equally, GSPO eliminates the instability factor caused by the unequal, fluctuating token-level weights in GRPO.
Also Read:
- Targeted Adjustments: How Reinforcement Learning Sparsely Fine-Tunes Large Language Models
- Optimizing Large Language Model Training Through Dynamic Data Weighting
Remarkable Benefits and Empirical Evidence
The empirical evaluations presented in the paper demonstrate GSPO’s significant advantages:
- Superior Training Stability: GSPO maintains stable training throughout, even with increasing computational investment and longer generation lengths. Crucially, it inherently resolves the expert-activation volatility issue in MoE models, completely removing the need for complex workarounds like Routing Replay. This simplifies training and allows MoE models to leverage their full capacity.
- Enhanced Efficiency and Performance: GSPO achieves better training accuracy and benchmark performance (on tasks like AIME’24, LiveCodeBench, and CodeForces) using the same amount of training compute and queries as GRPO. This indicates a more reliable and effective learning signal.
- Counter-Intuitive Clipping Observation: Interestingly, GSPO clips a much larger fraction of tokens (two orders of magnitude more than GRPO), yet it still achieves higher training efficiency. This suggests that GRPO’s token-level gradient estimates are inherently noisy and inefficient, while GSPO’s sequence-level approach provides a more robust signal, even with more aggressive clipping of less relevant samples.
- Simplified RL Infrastructure: GSPO’s reliance on sequence-level likelihoods makes it more tolerant of precision discrepancies between training and inference engines. This opens up the possibility of directly using likelihoods from inference engines for optimization, potentially avoiding costly recomputation steps and streamlining RL infrastructure, especially in scenarios like multi-turn RL.
The merits of GSPO have already contributed to the remarkable improvements seen in the latest Qwen3 models, proving its efficacy in unlocking the full potential of large-scale RL training for language models. The researchers envision GSPO as a robust and scalable foundation for the continued advancement of RL in LLMs.
