TLDR: Self-Rewarding PPO (SRPPO) is a novel method for aligning large language models (LLMs) that combines supervised fine-tuning (SFT) with proximal policy optimization (PPO). It introduces a ‘coherent reward’ derived from the SFT and pretrained models, eliminating the need for human preference annotations or explicit reward models. This allows for effective on-policy fine-tuning, significantly improving LLM generalization, data efficiency, and robustness across various NLP tasks using only demonstration data.
Large Language Models (LLMs) have transformed various fields, from generating creative text to complex problem-solving. However, ensuring these powerful models behave safely and usefully, a process known as alignment, remains a significant challenge. Traditionally, alignment involves two main stages: supervised fine-tuning (SFT) on expert demonstrations and reinforcement learning from human feedback (RLHF) using preference data.
SFT, while crucial, often faces limitations. It’s an ‘off-policy’ approach, similar to behavior cloning, meaning it learns directly from a fixed set of examples. This can lead to overfitting, where the model becomes too specialized to its training data and performs poorly on new, unseen tasks, especially when high-quality data is scarce. This is a common hurdle in developing robust and versatile LLMs.
On the other hand, ‘on-policy’ methods like Proximal Policy Optimization (PPO) have shown great success in the second stage of alignment (RLHF). PPO generates diverse samples during training, allowing the model to adapt and generalize better. The key challenge for applying PPO earlier in the alignment process, specifically with demonstration data, has been the absence of a clear reward signal.
Addressing this, a new research paper titled “Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only” introduces a novel fine-tuning method called Self-Rewarding PPO (SRPPO). This approach cleverly combines the strengths of SFT and PPO to achieve more effective and robust alignment using only demonstration data, without needing additional human preference annotations. You can read the full paper here: Self-Rewarding PPO Paper.
At the heart of SRPPO is a unique concept called the ‘coherent reward’. This reward function is designed as the log policy ratio between the SFT model (the model initially fine-tuned on demonstrations) and the pretrained base model. Essentially, it uses the pretrained model as a baseline and the SFT model as a target, quantifying how much the SFT model’s behavior diverges from the original. This divergence then serves as an implicit reward signal, guiding the model’s further refinement.
The SRPPO method unfolds in two sequential stages. First, a pretrained base model undergoes standard supervised fine-tuning (SFT) using high-quality demonstrations. This initial step establishes a foundational alignment. Second, the coherent reward, derived from this SFT model and the original pretrained model, is used to guide a subsequent RL fine-tuning stage with PPO. This PPO stage continuously refines the model, leveraging on-policy data to enhance generalization.
A significant advantage of SRPPO is that it eliminates the need for training a separate reward model or collecting expensive human preference annotations. The coherent reward is self-derived, making the process more efficient and scalable. Furthermore, SRPPO allows for the inclusion of additional prompts beyond those in the initial demonstration dataset during the PPO step. This is particularly beneficial in scenarios where obtaining a small amount of high-quality demonstration data is feasible, but generating more expert responses is difficult. The coherent reward effectively generalizes to these new prompts, further improving the model’s alignment.
Empirical evaluations using models like Mistral-7B and LLAMA3-8B demonstrated that SRPPO consistently outperforms traditional SFT methods and other alternatives across a range of natural language processing tasks. These tasks included instruction following, math reasoning, graduate-level question answering, and conversational ability. The results highlight SRPPO’s effectiveness in improving model alignment and generalization without requiring any new human-annotated data.
Also Read:
- Enhancing Mathematical Reasoning in Language Models: A Reinforcement Learning Approach to Budget Forcing
- RePULSe: A New Method to Drastically Reduce Undesirable Language Model Outputs
The research suggests that SRPPO offers a promising direction for LLM alignment, providing a scalable and robust framework that bridges the gap between supervised and reinforcement learning fine-tuning, ultimately leading to more capable and generalized language models.
