TLDR: A study compares Supervised Fine-tuning (SFT) and Reinforcement Fine-tuning (RFT) for continual post-training of multimodal large language models (MLLMs). It finds that SFT leads to catastrophic forgetting of both specific tasks and general knowledge, while RFT inherently preserves prior knowledge and even enhances general capabilities, achieving performance comparable to multi-task training without explicit forgetting mitigation strategies. This resilience is attributed to an implicit regularization mechanism in RFT. The paper also introduces Rollout-based Instance Filtering (RIF-RFT) to improve RFT’s efficiency and stability.
Foundation models, especially large language models that understand both text and images (multimodal large language models or MLLMs), are becoming increasingly important. These models need to constantly learn new information and adapt to evolving tasks. This process, known as continual post-training (CPT), is crucial for their real-world application. However, a major challenge in CPT is “catastrophic forgetting,” where models tend to forget previously learned information when adapting to new tasks.
A recent research paper, available at this link, delves into this problem by comparing two primary post-training methods: Supervised Fine-tuning (SFT) and Reinforcement Fine-tuning (RFT). The study investigates how these different learning approaches impact a model’s ability to retain knowledge during continuous learning.
The Problem with Supervised Fine-tuning (SFT)
Traditionally, SFT has been a common method for adapting models. In SFT, the model learns by being shown correct examples and adjusting its parameters to match those examples. However, the researchers found that when MLLMs undergo continual post-training using SFT, they suffer significantly from catastrophic forgetting. This means that as the model learns new tasks, its performance on older, previously learned tasks drops sharply. For instance, the paper highlights a substantial performance decrease on a task like ScienceQA after the model completes a sequence of other tasks. This forgetting isn’t just limited to specific tasks; SFT also severely degrades the model’s general knowledge and capabilities, even when all tasks are learned simultaneously (multi-task SFT).
The Promise of Reinforcement Fine-tuning (RFT)
In contrast, Reinforcement Fine-tuning (RFT) approaches the problem differently. Instead of being given correct answers, the model learns by generating its own responses and receiving feedback (rewards) on the quality of those responses. The study reveals that RFT methods are remarkably resilient to catastrophic forgetting. Models trained with RFT maintain strong performance on previously learned tasks even after adapting to new ones. Surprisingly, RFT can achieve performance comparable to multi-task training, where a model learns all tasks at once, without needing explicit strategies like data replay to prevent forgetting. Furthermore, RFT not only preserves but can even enhance the model’s general knowledge and abilities, such as its performance on benchmarks like MMMU and MMLU-Pro, and even reduces the tendency for “hallucinations” (generating incorrect or nonsensical information).
Why RFT Works: Implicit Regularization
To understand why RFT is so effective, the researchers conducted further analysis. They investigated whether common mechanisms like KL-divergence penalties (which prevent drastic changes to the model) or Chain-of-Thought (CoT) reasoning (where the model explains its steps) were the primary reasons for RFT’s stability. Their findings suggest that these explicit mechanisms are not the main drivers. Instead, the key factor is an “implicit regularization” inherent to RFT. This means that the way RFT updates the model’s parameters naturally makes it more conservative in areas important for old tasks. This conservatism is influenced by the variability of the reward signal, effectively acting as a built-in mechanism to prevent forgetting.
Improving RFT: Rollout-based Instance Filtering (RIF-RFT)
While RFT is powerful, its learning process can sometimes be inefficient, especially when the model struggles to generate good responses for certain training examples. To address this, the paper proposes a new method called Rollout-based Instance Filtering for RFT (RIF-RFT). This technique filters out “incompetent samples” – training examples for which the model consistently fails to produce useful responses. By focusing RFT on instances where it can receive a productive learning signal, RIF-RFT improves both the stability and efficiency of the training process without compromising its ability to protect knowledge. This allows for competitive performance while using significantly less training data.
Also Read:
- Beyond Benchmarks: How Fine-Tuning Methods Shape LLM General Intelligence
- Enhancing Language Models: A Unified Approach to Fine-Tuning
Conclusion
This research provides compelling evidence that Reinforcement Fine-tuning is a fundamentally more suitable paradigm for the continual adaptation of foundation models compared to traditional Supervised Fine-tuning. Its inherent ability to mitigate catastrophic forgetting and preserve general capabilities makes it a robust approach for developing models that can continuously learn and evolve in real-world scenarios.
