TLDR: Reinforcement learning for reasoning in large AI models often causes them to forget general skills. Researchers propose RECAP, a new training method that mixes general data and dynamically reweights learning objectives based on their convergence and stability. This approach successfully preserves broad capabilities while improving reasoning and making AI responses more concise.
Large Language Models (LLMs) and Vision-Language Models (VLMs) have made incredible strides in complex reasoning tasks. Techniques like Reinforcement Learning with Verifiable Rewards (RLVR) have been instrumental in pushing these models to excel in areas like mathematical problem-solving and instruction following. However, this specialized training comes with a significant challenge: the risk of models ‘forgetting’ their foundational, general-purpose skills.
This phenomenon, often referred to as capability regression or catastrophic forgetting, means that while an AI model becomes a whiz at reasoning, it might simultaneously lose proficiency in core abilities such as perception, factual accuracy, or even safety. Our research empirically confirms this concern, showing that many open-source reasoning models actually perform worse on general tasks like visual perception and robustness after being fine-tuned for reasoning.
Existing methods to combat this forgetting, such as KL divergence regularization or experience replay, have their limitations. KL divergence, for instance, is often calculated on the current task, meaning it doesn’t guarantee the preservation of broader knowledge. Experience replay, which involves revisiting old data, becomes complicated when dealing with diverse types of information, making it hard to decide how much focus each learning objective should receive.
To tackle this critical issue, we introduce RECAP—Replay-Enhanced CApability Preservation. RECAP is a novel replay strategy that incorporates a dynamic objective reweighting mechanism designed to preserve general knowledge during the post-training phase of large reasoning models. Think of it as a smart tutor for AI, ensuring it doesn’t neglect its basic education while mastering advanced subjects.
Here’s how RECAP works: Alongside the specific reasoning tasks, RECAP samples data from general domains. It then continuously monitors the learning process for each objective—whether it’s a reasoning reward or a general knowledge loss. By observing how quickly an objective is converging and how stable its learning progress is, RECAP dynamically adjusts its training focus. If an objective, like adhering to a specific output format, is quickly mastered and becomes stable, RECAP reduces its weight. This allows the model to dedicate more learning capacity to harder, more volatile objectives, such as improving reasoning accuracy, without over-optimizing the easier ones.
This method is entirely end-to-end and can be easily integrated into existing RLVR pipelines without the need for training additional models or extensive fine-tuning. We conducted extensive experiments using benchmarks based on Qwen2.5-VL-3B and Qwen2.5-VL-7B models. Our results demonstrate RECAP’s effectiveness: it not only successfully preserves the general capabilities of these models but also enhances their reasoning performance by enabling more flexible trade-offs among different in-task rewards.
A particularly interesting finding from our studies is that RECAP encourages the models to generate shorter, more concise rationales for their answers, especially on non-mathematical tasks. This means the models become more efficient, reducing latency and computational costs, all without compromising the quality of their problem-solving abilities. For a deeper dive into our methodology and findings, you can read the full research paper here.
Also Read:
- Unlocking Smarter LLM Reasoning: How Internal Confidence Guides Learning
- Enhancing Mathematical Reasoning in Language Models: A Reinforcement Learning Approach to Budget Forcing
In conclusion, RECAP offers a principled and practical solution to the challenge of general capabilities forgetting in large reasoning models. By intelligently balancing specialized reasoning gains with the preservation of broad knowledge, RECAP helps create more robust, efficient, and truly intelligent AI systems.
