TLDR: A new research paper introduces a “two-stage assumption” for aligning language models (LLMs) with human preferences. It posits that early in the alignment process (preference injection stage), diverse data is most effective, while later (preference fine-tuning stage), high-quality data is crucial. The paper presents a boundary measurement algorithm to identify which stage a model is in, supported by empirical evidence across various LLMs and theoretical analysis, offering a systematic approach to optimizing data selection for LLM alignment.
Aligning large language models (LLMs) with human preferences is a crucial step in building reliable artificial intelligence systems. This process, often framed as maximizing a reward that reflects human choices, has seen methods like Direct Preference Optimization (DPO) gain prominence. DPO directly optimizes the model from existing preference data, and recent advancements have even incorporated ‘on-policy’ sampling, where new preference candidates are generated during the training process itself.
However, a new research paper titled “Diversity First, Quality Later: A Two-Stage Assumption for Language Model Alignment” by Zetian Sun, Dongfang Li, and Baotian Hu from Harbin Institute of Technology (Shenzhen) reveals a fascinating insight: on-policy data isn’t always superior. Their findings show a systematic difference in effectiveness between static (off-policy) and dynamically generated (on-policy) preference data, with some models benefiting significantly from on-policy data (like Llama-3 with a 3x effectiveness increase) while others perform worse (like Zephyr with a 0.4x effectiveness).
The Two-Stage Alignment Assumption
To explain this phenomenon, the researchers propose a novel “alignment stage assumption,” which divides the language model alignment process into two distinct phases:
- Preference Injection Stage: This initial stage benefits most from diverse data.
- Preference Fine-tuning Stage: This later stage thrives on high-quality data.
Through extensive theoretical and empirical analysis, the paper characterizes these stages and introduces an effective algorithm to identify the boundary between them. Experiments conducted on five different language models (Llama, Zephyr, Phi-2, Qwen, Pythia) and two alignment methods (DPO, SLiC-HF) demonstrate the general applicability of this two-stage assumption and the boundary measurement.
Why Diversity First, Quality Later?
The research suggests that in the early “preference injection stage,” models are still developing a foundational understanding of human preferences. At this point, a wide variety of data, even if not perfectly high-quality, helps the model explore the vast landscape of possible responses and inject a broad range of preference knowledge. This diversity helps the model better approximate the ‘ground-truth’ preference distribution.
As the model progresses and enters the “preference fine-tuning stage,” it has already absorbed a good deal of preference knowledge. The goal then shifts from broad exploration to refining its responses within high-reward regions. Here, high-quality data becomes paramount, allowing the model to hone its ability to generate truly preferred outputs.
Identifying the Boundary
A key contribution of this work is the “boundary measurement algorithm.” This algorithm helps determine which stage a model is currently in by comparing how well the model’s generated preference candidates (on-policy) and static preference candidates (off-policy) align with a ‘ground-truth’ preference model. If off-policy data (which tends to be more diverse) is more effective, the model is likely in the preference injection stage. If on-policy data (which tends to be higher quality as the model improves) is more effective, it’s in the fine-tuning stage.
The empirical results strongly support this. For instance, Llama-3 consistently showed characteristics of being in the preference fine-tuning stage, benefiting more from on-policy data. Phi-2, on the other hand, behaved like it was in the preference injection stage, performing better with off-policy data. Zephyr demonstrated a transition, moving from the injection stage to the fine-tuning stage after initial training with off-policy data.
Also Read:
- Evaluating LLM Alignment: A New Framework for Safety and Performance
- Tailoring AI: A New Approach to Aligning Language Models with Diverse Human Values
Broader Implications
This research provides a systematic and methodological framework for understanding language model alignment. It offers actionable insights for researchers and practitioners, guiding them on how to synthesize preference data that is both efficient and effective for training policy models. By understanding these distinct stages, it’s possible to optimize the data selection process, leading to more robust and human-aligned LLMs. For more details, you can read the full paper here.
While this two-stage assumption simplifies a complex process, it offers a valuable abstraction that can significantly improve alignment strategies. Future work could explore the influences of reward over-optimization and sample efficiency within this framework.
