TLDR: A new research paper introduces a sample-wise, chance-adjusted metric to map how post-training affects knowledge retention in large language models (LLMs). It finds that catastrophic forgetting is not as widespread as commonly believed, with effects varying significantly based on the post-training method and scale. Domain-continual pretraining causes moderate forgetting, while instruction tuning and SFT/RL from base models show low-to-moderate forgetting but significant backward transfer in math and logic. Reasoning training from instruct models has mixed results depending on data scale, and model merging did not reliably mitigate forgetting in their study. The findings offer a clearer understanding of knowledge dynamics in LLMs, guiding future development towards more generally capable AI.
Large Language Models (LLMs) are at the forefront of AI innovation, constantly evolving through various training stages. While initial pre-training equips them with vast knowledge, subsequent ‘post-training’ steps, like instruction tuning or reinforcement learning, are crucial for enhancing specific capabilities such as coding, math, and safety. However, a long-standing concern in AI is ‘catastrophic forgetting’ – the tendency for models to lose previously learned information when acquiring new skills.
A recent research paper, titled “Mapping Post-Training Forgetting in Language Models at Scale,” by Jackson Harmon, Andreas Hochlehnert, Matthias Bethge, and Ameya Prabhu, delves into this critical issue. Their work challenges the common assumption that sequential training inevitably leads to significant knowledge erosion in LLMs. Instead, they reveal a more nuanced picture, showing that forgetting is not uniform and heavily depends on the post-training method and its scale.
A New Way to Measure Forgetting
Traditional methods for evaluating forgetting often compare overall accuracy before and after training. This approach can be misleading because it treats all knowledge as interchangeable. For instance, forgetting one historical fact might appear to be offset by remembering another, even if both pieces of information are uniquely valuable. To address this, the researchers propose a novel ‘sample-wise’ paradigm.
This new metric focuses on individual pieces of knowledge. They define ‘forgetting’ as instances where a model correctly answers a question before post-training but gets it wrong afterward (a 1→0 transition). Conversely, ‘backward transfer’ refers to cases where a model initially answers incorrectly but then gets it right after post-training (a 0→1 transition). Furthermore, for multiple-choice questions, they introduce ‘chance-adjusted’ variants of these metrics. This adjustment accounts for the possibility of random guessing, ensuring that the measured forgetting or transfer reflects genuine changes in the model’s knowledge rather than lucky or unlucky guesses.
Also Read:
- Mitigating Forgetting in Language Models Through Selective Memory Finetuning
- AI’s Internal Struggle: Why Large Reasoning Models Overlook Instructions During Complex Thought
Key Insights from Large-Scale Analysis
The study applied this framework across various post-training stages, model sizes, and data scales, leading to several significant findings:
-
Domain-Continual Pretraining: When LLMs are continuously pre-trained on specific domains (like coding or math), it causes moderate forgetting across general knowledge categories. However, the backward transfer (gains in non-target tasks) is limited. Interestingly, larger models tend to forget less in this scenario.
-
Instruction Tuning and SFT/RL from Base Models: Instruction tuning, which helps models interact more naturally with users, generally leads to low-to-moderate forgetting. There are some spikes in ‘Culture’ and ‘Knowledge’ categories. Crucially, these methods show moderate-to-high backward transfer, particularly in ‘Math’ and ‘Logic’ abilities. As model parameters increase, both forgetting and backward transfer tend to decrease.
-
Reasoning Training (SFT/RL) from Instruction-Tuned Models: The impact of training models for explicit reasoning (like using a ‘scratchpad’ to think step-by-step) varies significantly with the amount of data used. In low-data scenarios, both forgetting and backward transfer are minimal. However, in high-data regimes, the effects are mixed, with no single factor consistently explaining the dynamics.
-
Model Merging: The study also explored whether combining different models (model merging) could help mitigate forgetting. Their findings suggest that, in their experimental setup, model merging does not reliably reduce forgetting. This indicates that while promising, more research is needed to understand its effectiveness in this context.
Overall, the research indicates that catastrophic forgetting in modern LLM post-training pipelines is less severe than often assumed in continual learning literature. The extent of forgetting is highly dependent on the specific post-training method and its scale.
This framework provides a valuable tool for understanding how post-training alters the knowledge acquired during pre-training. It paves the way for developing AI systems that can accumulate new capabilities without sacrificing their broad general knowledge, moving closer to truly capable AI. For more details, you can read the full paper here.
The authors suggest future directions for preventing forgetting, including designing objectives that explicitly penalize knowledge loss, using targeted synthetic data to repair specific forgotten information, and integrating retrieval mechanisms to reduce reliance on in-model knowledge storage.
