TLDR: This research investigates how to teach Large Multimodal Models (LMMs) new skills without causing them to forget previously learned abilities. The study identifies that ‘forgetting’ is often a temporary shift in the model’s output distribution. It proposes two effective and simple fine-tuning methods: updating only the self-attention projection layers or only the MLP Gate&Up layers. These methods achieve significant learning on new tasks while largely preserving existing knowledge across different LMM architectures.
Large Multimodal Models (LMMs) are powerful AI systems capable of understanding and generating responses based on both images and text. Models like LLaVA and Qwen2.5-VL can perform a wide array of tasks, but when it comes to specialized domains or new skills, they often fall short. The challenge then becomes: how do we teach these advanced models new tricks without making them forget what they already know?
Retraining an LMM from scratch is an incredibly expensive and time-consuming endeavor, costing millions of dollars and weeks of effort. This makes finding efficient ways to update existing models a critical concern. Traditional fine-tuning, while common, often leads to what’s known as ‘catastrophic forgetting,’ where a model becomes an expert in the new task but loses its general abilities.
A recent study from the University of Illinois Urbana-Champaign delves into this very problem, exploring continual learning in LMMs. The researchers sequentially fine-tuned models on five diverse target skills, including bird classification, counting, medical visual question answering (VQA), optical character recognition (OCR) reading, and time reading. Simultaneously, they monitored the models’ general abilities across eight widely used benchmarks.
The findings revealed some surprising behaviors. Firstly, what appeared to be ‘forgetting’ on general tasks after narrow fine-tuning wasn’t always permanent; performance could partly recover at later stages. This led the researchers to investigate the underlying cause, tracing the behavior to a measurable shift in the model’s output token distribution.
To understand this shift, they used a simple ‘counting-bias probe.’ When a model was fine-tuned for a counting task, for example, tuning certain components significantly increased the likelihood of the model outputting numeric tokens, even for tasks that didn’t require counting. This bias was strongly correlated with the observed drop in performance on general held-out tasks.
Guided by this insight, the study identified two simple yet robust fine-tuning strategies that effectively teach new skills while limiting this output distribution drift:
Also Read:
- Enhancing Multimodal Models for Complex Object Descriptions with Chain-of-Thought Reasoning
- HARMO: A Hybrid Approach to Training Smarter Multimodal AI
Targeted Tuning Strategies
- Updating only the **self-attention projection layers (SA Proj.)** within the language model. This approach delivered strong gains on target tasks with minimal forgetting on held-out benchmarks.
- Updating only the **MLP Gate&Up layers** while freezing the Down projection in the Multi-Layer Perceptron (MLP) component. This method also achieved excellent learning on new tasks with very limited degradation of prior abilities.
These selective tuning choices proved effective across different LMM families, including LLaVA-OneVision, LLaVA-NeXT, and Qwen2.5-VL. The SA Proj. method was particularly stable, sometimes even showing slight improvements on held-out tasks, while MLP (Gate&Up) offered a near-maximal learning capability with controlled forgetting.
The research also compared these strategies to other common forgetting mitigation techniques like Learning without Forgetting (LwF), Low-Rank Adaptation (LoRA), Mixture-of-Experts (MoE), and weight-space ensembling (WiSE-FT). The selective tuning of SA Proj. or MLP Gate&Up matched or even surpassed these methods in balancing learning and stability, all while being simpler to implement without extra modules or complex replay mechanisms.
In conclusion, this work provides a deeper understanding of how LMMs learn and forget, demonstrating that apparent forgetting is often a temporary output distribution shift rather than a permanent loss of knowledge. The proposed tuning recipes offer practical and efficient ways to continuously improve LMMs, reducing the significant environmental and financial costs associated with model adaptation. For more details, you can read the full paper here.
