TLDR: GCPO (Group Contrastive Policy Optimization) is a novel reinforcement learning algorithm designed to improve the reasoning capabilities of large language models (LLMs). It addresses the limitations of existing methods by incorporating “Golden Answers” (GAs) – external, correct reference solutions – into the training process. This allows the model to receive clear guidance, especially when it struggles to generate correct responses, thereby boosting training efficiency and enabling smaller models to learn advanced reasoning patterns from more powerful sources. GCPO also utilizes sequence-level importance sampling and removes KL divergence, leading to superior performance across various math benchmarks compared to baseline models and other advanced techniques like DAPO.
Reinforcement learning (RL) has become a powerful tool for improving the reasoning abilities of large language models (LLMs). A key challenge in this area is extending the capabilities of smaller models. However, existing algorithms, such as Group Relative Policy Optimization (GRPO), face a significant limitation: their learning is entirely dependent on the model’s own generated responses. This means they struggle to learn from situations where all generated answers are either completely wrong or completely correct, leading to inefficiencies and missed learning opportunities.
Introducing GCPO: Going for Gold
A new method called Group Contrastive Policy Optimization (GCPO) addresses this fundamental problem by introducing external, standard reference answers, termed “Golden Answers” (GAs). When a model is unable to solve a problem on its own, a Golden Answer provides the correct response, effectively steering the model towards an accurate update direction. This innovative approach offers two main benefits: it significantly improves training efficiency by making full use of every training sample, and it enables smaller models to emulate the problem-solving strategies of these reference answers, thereby enhancing their generalization in reasoning tasks.
The core idea behind GCPO is to provide a clear, unambiguous positive example for the model to learn from. In situations where a model’s generated responses for a question are all incorrect, GCPO replaces one of these failed responses with a Golden Answer. This Golden Answer can be the true ground-truth solution or a high-quality response generated by a more powerful language model. This ensures that even when a problem is intrinsically difficult for the model, it still receives clear guidance on the correct path forward.
How GCPO Works
GCPO builds upon the principles of GRPO but introduces several crucial modifications. Firstly, it tackles the issue of vanishing policy gradients that occur when all rollouts are either incorrect or correct. By injecting a Golden Answer, GCPO ensures there’s always a positive example, providing a clear direction for optimization and improving learning efficiency.
Secondly, GCPO refines the importance sampling mechanism. Traditional methods often apply importance sampling at the token level, which can be suboptimal when rewards are based on the entire sequence of generated text, such as a multi-step mathematical solution. GCPO shifts to sequence-level importance sampling, aligning the reward signals more consistently with the policy optimization. This seemingly minor change has been shown to enhance both performance and training stability.
Lastly, GCPO, similar to other advanced methods like DAPO, omits the KL divergence penalty from its objective function. The reasoning is that for complex reasoning tasks, a model’s optimal output distribution might diverge significantly from its initial baseline. Imposing a KL constraint would hinder these necessary shifts and limit the model’s ability to break through its existing performance boundaries.
Also Read:
- Empowering Language Models: How TAPO Integrates Reasoning and Adaptive Tool Use
- Stratified GRPO: Enhancing LLM Search Agents by Addressing Trajectory Diversity
Outstanding Results and Future Potential
Extensive experiments were conducted using the DeepSeek-R1-Distill-Qwen-1.5B model across 10 different math benchmarks, including AIME2024, GSM8K, and MATH. The results demonstrate that GCPO consistently outperforms both the baseline model and DAPO across nearly all datasets. For instance, on the AIME 2024 dataset, GCPO achieved a 25% improvement over DAPO, and a remarkable 54% performance gain over the baseline model on the MathQA dataset. These substantial improvements highlight the robustness and effectiveness of the proposed method.
An ablation study further confirmed the impact of GCPO’s design choices. It showed that sequence-level importance sampling is indeed more beneficial than token-level sampling, and that removing the KL divergence penalty leads to better performance. These findings underscore how GCPO’s strategic modifications contribute to more stable and efficient training.
While GCPO currently focuses on mathematical tasks and requires the collection of Golden Answers for the training set, its principles are broadly applicable. The researchers hope that GCPO will inspire new ideas in model inference and lead to fundamental advancements in training larger-scale models, potentially integrating with tool-using Chain-of-Thought frameworks for a wider range of problems. For more technical details, you can refer to the full research paper here.
