TLDR: The paper provides a clear, from-scratch explanation of key reinforcement learning (RL) algorithms for instruction tuning large language models (LLMs), including SFT, Rejection Sampling, REINFORCE, TRPO, PPO, GRPO, and DPO. It simplifies complex concepts for LLM practitioners, reviews emerging RLMT techniques like RLAIF and Process Supervision, and introduces GRAPE (Generalized Relative Advantage Policy Evolution)—a new framework that combines RLHF and RLAIF elements, uses category-specific rubrics with AI-generated critiques and confidence scores, and eliminates the need for separate value or reward models for more efficient and flexible LLM alignment.
A new research paper, “Understanding Reinforcement Learning for Model Training, and future directions with GRAPE,” by Rohit Patel, offers a comprehensive guide to the complex world of training large language models (LLMs) using reinforcement learning. The paper aims to make these advanced techniques accessible, breaking down key algorithms step-by-step with a focus on LLMs, and then introduces a novel framework called GRAPE (Generalized Relative Advantage Policy Evolution).
From Pre-trained to Instruction-Tuned: The Foundation
Initially, LLMs are “pre-trained” to predict the next word in a sequence, making them excellent at auto-completion. However, for them to be truly useful and follow instructions, they need “instruction tuning.” This process teaches models to respond in a question-answer format, follow specific instructions, and avoid generating harmful or nonsensical content. The paper explains how special “delimiter” tokens like <USER> and <MODEL> help the model understand when it’s being asked a question and when it should provide an answer.
The simplest form of instruction tuning is Supervised Fine-Tuning (SFT). This involves training the model on a smaller, high-quality dataset of question-answer pairs. The model learns by minimizing a “negative log-likelihood” loss function, essentially trying to maximize the probability of generating the correct next token in the answer. SFT is a crucial first step, but creating large, diverse datasets for it can be challenging.
To scale up data creation, Rejection Sampling was introduced. Here, the model generates multiple responses to a prompt, and humans (or another AI) select the best one. This chosen response is then used for SFT. While it reduces manual effort, rejection sampling has drawbacks: it’s a discrete learning process, the model doesn’t learn from its “mistakes” (rejected answers), it’s computationally expensive, and it risks “model collapse” where the model becomes overly optimized for a narrow set of criteria, losing diversity or coherence.
Reinforcement Learning: Learning from Feedback
To overcome the limitations of rejection sampling, reinforcement learning (RL) methods are employed. In RL, the LLM is seen as a “policy” that takes “actions” (generating tokens) in a “state” (the current text). The goal is to maximize a “reward” signal.
REINFORCE is one of the foundational RL algorithms. Instead of just using the single best response, REINFORCE uses all generated responses, weighting them by a “reward function” that scores how good each response is. To make this process more stable, a “baseline” is subtracted from the reward, leading to an “advantage function” that measures how much better a specific token choice is compared to what the model would typically do. A “reward model” is trained separately using human preferences to provide these scores.
A critical challenge in RL is preventing “model collapse” or overfitting. Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) address this by ensuring that the updated model’s behavior doesn’t stray too far from its previous version. They achieve this by adding a penalty based on the “KL divergence,” a measure of distance between probability distributions, or by “clipping” the ratio of new to old probabilities in the loss function (PPO). This keeps the model’s updates within a “trust region,” promoting more stable learning.
Group Relative Policy Optimization (GRPO) is a variation of PPO that simplifies the advantage function. Instead of needing a separate “value model” to calculate the baseline, GRPO calculates the advantage of a response by comparing its reward to the average reward of other responses generated for the same question within a group. This makes it more efficient by eliminating the need for a value function.
Further simplifying the process, Direct Preference Optimization (DPO) bypasses the need for explicit reward and value models altogether. It directly optimizes the language model using human preference data (e.g., “response A is better than response B”). DPO reformulates the problem so that the model learns to increase the probability of preferred responses and decrease the probability of dispreferred ones, while still implicitly incorporating a KL penalty to prevent model collapse.
New Frontiers in LLM Alignment
The paper also reviews several cutting-edge approaches that are shaping the future of LLM training:
- Curriculum Learning: Training models by starting with easier tasks and gradually moving to more complex ones, which can lead to faster learning and better performance.
- Reinforcement Learning with AI Feedback (RLAIF): Replacing human data labelers with AI models to provide feedback. This offers high scalability but introduces new challenges related to AI bias and the quality of the “constitution” (principles) guiding the AI feedback.
- Process Supervision: For complex reasoning tasks (like math or coding), instead of just rewarding the final answer, models are supervised on their intermediate, step-by-step reasoning process. This ensures not just correct outcomes but also sound logic.
- Game Theory and Self-Play: Training models through interactive scenarios, such as having AI agents debate each other or play against previous versions of themselves. This can generate powerful training signals and help models learn in domains where human supervision is difficult.
- Advanced Offline Policy Optimization: Developing more efficient RL algorithms that can learn effectively from static, pre-collected datasets, addressing limitations of preference-pair methods like DPO for certain tasks.
Also Read:
- Structuring Intelligence: Language Models Crafting Hierarchical Learning Environments for AI Agents
- Making Sense of AI Actions: TalkToAgent’s Approach to Explaining Reinforcement Learning
Introducing GRAPE: A Unified Framework
The paper concludes by introducing GRAPE (Generalized Relative Advantage Policy Evolution), a new framework designed to integrate many of these advancements. GRAPE aims to eliminate the need for separate value and reward models while still allowing for human feedback. Key features of GRAPE include:
- Category-Specific Rubrics: Questions are grouped into categories (e.g., coding, math, safety), each with its own detailed evaluation rubric.
- AI-Generated Critiques and Scores: For each rubric item, an AI model generates a critique, a score (0-1), and a confidence level.
- Weighted Aggregation: Scores from individual rubric items are aggregated into a single reward, weighted by their importance and the AI’s confidence.
- Advantage Function: Similar to GRPO, the advantage is calculated by comparing a response’s reward to the average reward within its group, removing the need for a value function.
- Flexibility and Iteration: GRAPE allows for continuous refinement of rubrics, generation strategies, and scoring models, and can seamlessly integrate human feedback and existing SFT data.
GRAPE represents a modular and transparent approach to model alignment, suggesting that the future of LLM training lies in making models better at fundamental reasoning, rather than just balancing various capabilities.
