TLDR: GTPO (Group-relative Trajectory-based Policy Optimization) is a new method for training Large Language Models (LLMs) that addresses key limitations of existing policy optimization techniques like GRPO. It tackles ‘gradient conflicts’ on shared tokens by selectively reweighting updates and prevents ‘policy collapse’ through entropy-based filtering and regularization. Experiments show GTPO leads to more stable training and improved performance on mathematical reasoning benchmarks without needing a reference model.
Large Language Models (LLMs) have become incredibly powerful, and a key part of their development involves optimizing them to align with desired human preferences. This is often done using policy-based optimization techniques. One of the more recent and effective approaches in this area is Group-relative Policy Optimization (GRPO).
However, a new research paper titled “GTPO: Trajectory-Based Policy Optimization in Large Language Models” by Marco Simoni, Aleksandar Fontana, Giulio Rossolini, and Andrea Saracino, identifies and analyzes two significant limitations of GRPO. First, tokens that appear frequently in completions with both positive and negative rewards can lead to conflicting updates. This can reduce their output probability, even if they are crucial for maintaining proper structure, such as formatting tags. Second, negatively rewarded completions can sometimes penalize confident responses, shifting the model towards less likely tokens. This progressively flattens the output distribution and can degrade the learning process, a phenomenon the authors refer to as ‘policy collapse’.
To tackle these challenges and provide a more stable and effective policy optimization strategy, the researchers introduce GTPO, or Group-relative Trajectory-based Policy Optimization. GTPO approaches the sequence of generated tokens as a ‘trajectory’ of decisions made by the LLM. Its core idea is to prevent unwanted divergence among these trajectories within the same group, thereby enhancing stability and rewards.
How GTPO Works
GTPO incorporates two main components to achieve its goals:
Conflict-Aware Gradient Correction: GTPO identifies what it calls ‘conflict tokens’. These are tokens that appear in the same position across different completions that have received opposite rewards (some positive, some negative). For these conflict tokens, GTPO implements a clever correction mechanism: it skips negative updates, effectively preventing the model from being penalized for these essential tokens, while simultaneously amplifying positive updates. This is particularly beneficial for formatting tokens or structural elements like ‘<reasoning>’ or ‘</answer>’, which are vital for the answer’s structure but can often be caught in gradient conflicts.
Entropy-Based Policy Regularization: To prevent the dreaded policy collapse, GTPO employs entropy-based regularization. The paper notes that GRPO’s standard KL-divergence regularization often reacts too slowly to prevent this collapse, whereas monitoring the entropy of the LLM’s outputs provides a clearer, real-time signal of instability. GTPO addresses this by filtering out completions whose entropy exceeds a certain threshold, especially for models that naturally tend to produce low-entropy outputs. Additionally, it applies a regularization term that penalizes high-entropy behavior, encouraging the model to maintain a more focused and stable output distribution. A significant advantage of GTPO is that, unlike GRPO, it does not require a separate reference model during training, making the process more efficient and lightweight.
Also Read:
- Enhancing LLM Reasoning with Consistency-Aware Policy Optimization
- Fine-Grained Reward Signals for Large Language Models
Experimental Validation
The effectiveness of GTPO was validated through extensive experiments on well-known benchmarks such as GSM8K, MATH, and AIME 2024, using LLaMA-8B and Qwen 2.5-3B models. The results consistently showed that GTPO outperforms GRPO and standard Supervised Fine-Tuning (SFT) across various settings. GTPO demonstrated more stable training dynamics, avoiding the policy collapse observed in GRPO, especially on more challenging datasets like MATH. It also achieved higher accuracy and better formatting rewards. Furthermore, GTPO showed stronger out-of-distribution generalization, performing better on the AIME2024 dataset compared to both GRPO and SFT.
Ablation studies within the paper further highlighted the critical role of both the entropy-based filtering and the conflict-aware gradient correction components in maintaining training stability and improving performance. The research suggests that GTPO offers a robust and effective policy optimization method for training and aligning large language models, leading to more stable and accurate outputs.
For more in-depth technical details, you can read the full research paper available here.
