TLDR: The paper theoretically and empirically proves that the GRPO reinforcement learning algorithm inherently creates a process reward model (PRM) that assigns rewards to intermediate steps, even when trained with only outcome-level rewards. It identifies a flaw in GRPO’s objective related to how these implicit step-level rewards are weighted, which can impede learning. The authors propose a simple modification, λ-GRPO, which normalizes these step contributions. Experiments show λ-GRPO significantly improves training speed, validation accuracy, and performance on reasoning tasks, suggesting that GRPO’s built-in PRM can be leveraged to boost performance without needing costly, explicit PRMs.
In the rapidly evolving field of artificial intelligence, particularly in the training of large language models (LLMs), reinforcement learning (RL) algorithms play a crucial role. One such algorithm, Group Relative Policy Optimization (GRPO), has gained traction for its simplified approach to RL training, notably by doing away with complex components like the critic model and generalized advantage estimation (GAE) found in other methods like Proximal Policy Optimization (PPO).
A recent research paper, titled “GRPO IS SECRETLY A PROCESS REWARD MODEL” by Michael Sullivan, delves into the inner workings of GRPO, revealing a fascinating and previously unacknowledged aspect of its design. The paper posits that GRPO, under certain conditions, inherently induces what is known as a Process Reward Model (PRM).
Understanding Process Reward Models (PRMs)
To appreciate this discovery, it’s important to understand PRMs. Unlike outcome reward models (ORMs) which assign a single reward at the end of a task, PRMs provide rewards to intermediate steps within a multi-step process. This finer-grained feedback is particularly beneficial for tasks requiring sequential reasoning, such as mathematical problem-solving, as it helps guide the model through each stage of its thought process.
Traditionally, training neural PRMs has been a costly endeavor, requiring extensive human annotation for step-level rewards and being susceptible to issues like reward hacking. This has led to the development of heuristic or Monte-Carlo-based PRMs that don’t rely on learned parameters or intensive human labeling.
GRPO’s Hidden PRM
The core finding of Sullivan’s paper is a theoretical proof, backed by empirical evidence, that GRPO implicitly generates a non-trivial PRM. This means that even when GRPO is given only a final outcome reward, it internally assigns and optimizes for rewards at various intermediate steps of a generated sequence. This occurs when subsets of trajectories within a training group share identical prefixes, a condition the paper empirically shows is met very frequently in real-world scenarios.
Essentially, GRPO is not just looking at the final result; it’s also evaluating the journey, assigning step-level rewards and advantages based on the mean outcome-level reward of trajectories that share common initial segments. This covert mechanism provides a rich, structured step-level reward signal that was previously overlooked.
Identifying a Flaw and Proposing a Solution
Further investigation into GRPO’s hidden PRM structure revealed a critical flaw in its objective function. The paper highlights that the contribution of each process set (a group of trajectories sharing a common prefix) to the overall loss is scaled by its size. This non-uniform distribution of process steps can hinder both exploration (when a process set has a positive advantage but is heavily scaled, discouraging exploration of alternatives) and exploitation (when a process set has a negative advantage and is heavily scaled, reducing the likelihood of generating high-reward trajectories within that set).
To mitigate this defect, the paper proposes a simple modification to the algorithm called λ-GRPO. This enhanced version introduces a PRM-aware normalization factor that scales the token-level loss by the inverse of the process set size. This ensures that each process set contributes equally to the loss, addressing the imbalance.
Impressive Results
The empirical evaluation of λ-GRPO demonstrated significant improvements over standard GRPO. LLMs trained with λ-GRPO achieved higher validation accuracy and reached peak performance more rapidly, often with a training speedup of approximately 2x. On various downstream reasoning benchmarks, λ-GRPO consistently outperformed its standard counterpart, showcasing its superiority.
Crucially, these performance gains come with a negligible impact on training time and cost, as λ-GRPO merely leverages and refines the existing, implicit PRM structure within GRPO, rather than requiring the generation of new, explicit step-level reward signals.
Also Read:
- Optimizing LLM Agent Training with Tree-Based Reinforcement Learning
- EvA-RL: Training Reinforcement Learning Policies for Easier and More Accurate Evaluation
Implications for Future RL Training
The findings of this research challenge the conventional wisdom that costly, explicitly-defined PRMs are always necessary for GRPO. Instead, the paper suggests that future work could greatly benefit from exploiting the rich, built-in PRM structure already present in the vanilla GRPO algorithm. By understanding and appropriately modifying this inherent mechanism, significant performance boosts can be achieved efficiently.
For a deeper dive into the theoretical proofs and experimental details, you can access the full research paper here.
