TLDR: Contrastive Weak-to-Strong Generalization (ConG) is a new framework for scaling large language models (LLMs) by training stronger models on samples from aligned weaker ones. It addresses the limitations of traditional weak-to-strong methods, such as noise and biases in weak-model outputs, by leveraging the structural equivalence between implicit rewards and Contrastive Decoding. ConG generates higher-quality training data in two stages: first, using Contrastive Decoding for Supervised Fine-Tuning (ConG-S), and then refining the model with Direct Preference Optimization (ConG). Empirical results show consistent and significant improvements in robustness and generalization across different model families, while maintaining general capabilities on downstream tasks.
The field of Artificial General Intelligence (AGI) is constantly evolving, with researchers exploring innovative ways to scale the capabilities of large language models (LLMs). One promising approach is ‘weak-to-strong generalization,’ where powerful models are trained using data generated by less capable, but already aligned, models. This method aims to bypass the need for extensive human feedback or complex reward systems, which can be costly and time-consuming.
However, traditional weak-to-strong generalization faces significant hurdles. The outputs from weaker models often contain noise and biases, which can limit the robustness and overall effectiveness of the stronger models trained on this data. This challenge has prompted researchers to seek new methods for extracting higher-quality training signals from these weaker models.
A recent paper introduces a novel framework called Contrastive Weak-to-Strong Generalization (ConG), which offers a compelling solution to these problems. The core idea behind ConG is to leverage ‘implicit rewards’ and a decoding strategy known as ‘Contrastive Decoding’ (CD) to generate superior training samples. Implicit rewards approximate explicit rewards (like human feedback) by using the log-likelihood ratios between outputs from models before and after alignment. This provides a reliable signal for assessing the quality of generated text.
The researchers discovered a crucial connection: the structure of implicit rewards is mathematically equivalent to that of Contrastive Decoding. Contrastive Decoding is a technique that reduces noise in LLM generations by comparing probability distributions during the text generation process. This equivalence means that Contrastive Decoding can be interpreted as a method for generating responses that inherently maximize implicit reward, leading to higher-quality outputs.
Introducing ConG: A Two-Stage Framework
ConG builds on this fundamental equivalence and operates in two distinct stages:
1. ConG-S (Contrastive Decoding for Supervised Fine-Tuning): In this initial stage, the framework uses Contrastive Decoding between a pre-alignment weak model and a post-alignment weak model to generate high-quality responses. These ‘chosen’ samples are then used to perform Supervised Fine-Tuning (SFT) on the strong model. This process provides the strong model with a high-reward starting point, guiding its policy towards the desired preference distribution.
2. ConG (Generalization with Direct Preference Optimization): Following SFT, the strong model undergoes further refinement using Direct Preference Optimization (DPO). For each prompt, a Contrastive Decoding-generated response (from Stage I) is paired with an additional response sampled from the strong model after its SFT. The CD-generated response is considered the ‘preferred’ one due to its higher implicit reward. DPO then optimizes the strong model to maximize this reward gap, leading to more reliable and robust generalization.
Also Read:
- Enhancing Language Model Alignment with Adaptive Preference Optimization
- Boosting Language Model Performance with Contrastive Synthetic Data Generation
Empirical Validation and Key Findings
The researchers rigorously evaluated ConG across two major LLM families: Qwen2.5 and Llama3. The results consistently demonstrated that ConG significantly outperforms traditional weak-to-strong methods. On average, ConG yielded a substantial gain of about 16.5% over the base models, highlighting its effectiveness in improving capability transfer, denoising, and robustness.
Key observations from the experiments include:
- ConG showed significant improvements in both ‘self-alignment’ (where weak and strong models are the same) and ‘weak-to-strong alignment’ settings.
- The effectiveness of ConG was influenced by the ‘capability gap’ between weak and strong models. Smaller gaps generally led to larger improvements, and larger strong models were better at leveraging the preference signals from weak models.
- The ‘contrastive coefficient’ (alpha), a parameter in Contrastive Decoding, played a crucial role. Moderate values (e.g., 0.3 to 0.5) consistently produced the best alignment performance, balancing the enhancement of the preference signal without overwhelming the model’s original behavior.
- Importantly, ConG introduced negligible degradation on a diverse suite of downstream tasks, indicating that the method preserves the model’s general capabilities and overall utility.
While ConG presents a promising pathway toward AGI, the authors acknowledge some limitations, such as compatibility with mainstream inference acceleration techniques and the increased engineering burden of managing multiple weak model alignment states. Future work aims to address these challenges, including adapting Contrastive Decoding for faster inference and designing lighter-weight strategies for approximating weak-model alignment states.
This research marks a significant step forward in weak-to-strong generalization, offering a robust and effective framework for scaling LLMs without heavy reliance on human feedback. For more technical details, you can read the full paper here.
