TLDR: A new method called Correctness Relative Policy Optimization (CoRPO) has been introduced to fix a critical flaw in Group-relative Policy Optimization (GRPO) when training Large Language Models (LLMs) with non-binary (ordinal) rewards. GRPO’s traditional approach can accidentally reward incorrect behaviors, but CoRPO uses an adaptive baseline to ensure only correct actions are reinforced, leading to more stable learning and better generalization, especially in new, unseen tasks.
Large Language Models (LLMs) are becoming increasingly adept at complex tasks, thanks in part to sophisticated training methods like Reinforcement Learning (RL). One such method, Group-relative Policy Optimization (GRPO), has gained popularity for its simplicity and efficiency in fine-tuning LLMs for verifiable tasks like mathematics and code generation. However, new research from Anisha Garg and Ganesh Venkatesh at APPLIEDAI RESEARCH, CEREBRAS, highlights a critical flaw in GRPO when LLMs are trained using more nuanced, non-binary feedback, known as ordinal rewards.
The paper, titled “The Peril of Preference: Why GRPO fails on Ordinal Rewards,” explains that GRPO’s core mechanism—using a group’s average reward as a baseline—can inadvertently reinforce incorrect behaviors. When an LLM is still learning and produces many suboptimal solutions, the average reward across a group of attempts can be very low. In such scenarios, a solution that is merely “less bad” than others in the group might receive a positive learning signal, even if it’s objectively a failure. This means the model is encouraged to learn from mistakes, hindering its ability to achieve true correctness.
To address this significant challenge, the researchers introduce a novel formulation called Correctness Relative Policy Optimization (CoRPO). CoRPO is designed to overcome GRPO’s limitations by employing an adaptive baseline that ensures failed solutions are never positively reinforced. It achieves this through a clever two-phase approach:
CoRPO’s Adaptive Approach
Initially, when the LLM is performing poorly, CoRPO operates in a “Correctness-Seeking” phase. Here, the baseline is clamped at a predefined minimum correctness threshold. This guarantees that any trajectory falling below this quality bar receives negative reinforcement, effectively preventing the model from learning bad habits. This acts as a crucial safety rail, ensuring the model focuses on achieving fundamental correctness.
Once the policy consistently meets or exceeds this minimum quality threshold, CoRPO seamlessly transitions into a “Preference-Seeking” phase. In this stage, the baseline reverts to the standard group average, similar to GRPO. However, because the model is already producing acceptable solutions, this relative comparison now pushes the LLM to find optimal solutions rather than just “acceptable” ones. It fosters an “Aspirational Drive,” encouraging continuous improvement towards the best possible outcomes.
Also Read:
- Tool Zero: A New Path for LLMs to Master Tools Through Pure Reinforcement Learning
- Unlocking Advanced AI Reasoning for Creative Challenges
Empirical Validation and Impact
The researchers empirically validated CoRPO on a challenging code verification task, where an RL model was trained to assess the correctness of LLM-generated code. Their findings confirmed the flaw in GRPO, showing that a significant portion of failed trajectories indeed received positive advantages. In contrast, CoRPO demonstrated more stable convergence during training and significantly better generalization on out-of-domain tasks. This suggests that by enforcing a strict correctness guarantee, CoRPO helps LLMs learn a more robust and generalizable understanding of what constitutes a correct solution.
While CoRPO represents a critical step forward, the research also highlights new challenges, such as balancing the magnitude of learning updates to ensure efficient exploitation of learned knowledge while still encouraging exploration. Future work aims to refine CoRPO’s training schedule and explore even denser, per-step feedback mechanisms to further enhance LLM capabilities.
This work marks a significant advancement in enabling LLMs to learn genuinely new capabilities through reinforcement learning, moving beyond simple binary feedback to leverage richer, multi-dimensional information. For a deeper dive into the mathematical framework and empirical results, you can read the full research paper here.
