TLDR: CHORD is a novel framework that unifies Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) for Large Language Models (LLMs). It uses dynamic weighting with a global coefficient (µ) and a token-wise function (ϕ) to smoothly integrate expert knowledge while preserving the model’s exploratory capabilities, preventing issues like pattern disruption and overfitting, and leading to improved performance.
Large Language Models (LLMs) have become incredibly powerful, excelling in tasks from solving complex math problems to generating code. A significant part of their refinement comes from two key post-training methods: Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL).
SFT involves training LLMs on high-quality expert examples, helping them mimic desired response patterns. RL, on the other hand, encourages LLMs to explore and improve their behaviors by learning from direct feedback. While both are effective, combining them in traditional ways, like the common SFT-then-RL sequence, often presents challenges. Researchers have observed that this sequential approach can disrupt the model’s established knowledge and lead to overfitting to the expert data. This phenomenon is described as a “shift-readapt-overfit” progression, where the model initially struggles, then adapts, and finally becomes too specialized.
To overcome these limitations, a team of researchers from Alibaba Group has introduced a novel framework called CHORD. This framework offers a unified perspective on SFT and RL, integrating them not as separate stages but as dynamically weighted components within a single, continuous learning process. You can find the full research paper here: ON-POLICY RL MEETS OFF-POLICY EXPERTS : HARMONIZING SUPERVISED FINE -TUNING AND REINFORCEMENT LEARNING VIA DYNAMIC WEIGHTING.
CHORD employs a clever “dual-control mechanism” to manage the influence of expert data and ensure stable learning:
Global Coefficient (µ)
The first control is a global coefficient, µ, which acts as a dynamic balancer between RL and SFT. In traditional methods, the switch from SFT to RL is often abrupt. CHORD, however, starts with a high µ, giving more weight to learning from expert data (SFT). As training progresses, µ gradually decreases, smoothly shifting the focus towards on-policy exploration (RL). This gentle transition helps the model leverage expert knowledge without suffering from sudden disruptions or exposure bias, which can occur when models are trained solely on static expert data.
Also Read:
- Dynamic Self-Awareness for Efficient Large Language Model Reasoning
- Optimizing Data Mixtures for Language Models with Bayesian Approaches
Token-wise Weighting Function (ϕ)
The second, more fine-grained control is a token-wise weighting function, ϕ(·). This function modulates the learning signal for individual tokens (parts of words) in the expert data. Specifically, it assigns less weight to tokens that the model is already very certain about (either very high or very low probability). This prevents the model from becoming overly confident in existing patterns (which could limit exploration) and also avoids disruption from extremely improbable or divergent expert tokens. By focusing learning on tokens where the model is most “uncertain” (i.e., probabilities around 0.5), CHORD creates a “learning sweet spot” that encourages the model to absorb novel yet informative knowledge without destabilizing its core reasoning abilities.
Extensive experiments conducted on mathematical reasoning and general reasoning benchmarks demonstrate CHORD’s effectiveness. The framework consistently outperforms existing baselines, including the SFT-then-RL paradigm. CHORD achieves a more stable and efficient learning process, leading to significant improvements in model performance. It enables LLMs to selectively integrate beneficial reasoning patterns from expert data, such as incorporating verification steps in problem-solving, while still fostering their own exploratory capabilities and preventing overly lengthy or rigid responses. This balanced approach allows models to strategically absorb expert knowledge without stifling their inherent reasoning potential.
