TLDR: MCITlib is a new open-source library and benchmark designed to advance research in Multimodal Continual Learning (MCL) for Large Language Models. It provides standardized implementations of 8 algorithms and uses two carefully selected benchmarks to evaluate how well these models can continuously learn new information across different modalities (like vision and language) without forgetting old knowledge, addressing challenges like information leakage and inconsistent comparisons in the field.
Artificial intelligence systems are constantly evolving, much like human learning. A key challenge in this evolution is enabling AI to continuously acquire new knowledge and adapt without forgetting what it has already learned. This concept is known as continual learning. While traditional continual learning has seen success in single-modality tasks, the rise of Multimodal Large Language Models (MLLMs) has brought new complexities, as these models deal with multiple types of data, such as vision and language.
Multimodal Continual Learning (MCL) presents unique challenges. Beyond preventing catastrophic forgetting—where models lose previously learned information when acquiring new—MCL also requires handling intricate interactions between different data modalities and diverse task formats. To address these complexities and foster research in this critical area, a new comprehensive and evolving code library called MCITlib has been introduced.
Introducing MCITlib: A Unified Platform for Multimodal Continual Instruction Tuning
MCITlib serves as the first publicly available code library and benchmark specifically designed for continual instruction tuning of MLLMs. Its primary goal is to provide a standardized environment for researchers to compare and evaluate different MCL methods fairly. The library currently implements eight representative algorithms for Multimodal Continual Instruction Tuning, including methods like LoRA-FT, O-LoRA, MoELoRA, ModalPrompt, CL-MoE, HiDe-LLaVA, SEFE, and DISCO.
The development of MCITlib involved careful consideration of existing limitations in the field. Previous research often suffered from information leakage, where benchmark datasets overlapped with the pre-training data of MLLMs, leading to unfair evaluations. Additionally, a lack of consistent settings across different studies made it difficult to compare methods accurately. MCITlib tackles these issues by selecting benchmarks that minimize information leakage and by reproducing multiple methods under unified protocols.
Rigorous Benchmarking and Evaluation
To ensure robust and fair comparisons, MCITlib utilizes LLaVA-1.5-7b as its base model and employs Parameter-Efficient Fine-Tuning (PEFT) strategies. The training process follows a rehearsal-free continual learning setting, meaning data from previous tasks is not reused when learning new ones.
Two carefully selected benchmarks are central to MCITlib’s evaluation: the UCIT benchmark and the MLLM-DCL benchmark. The UCIT benchmark includes six diverse datasets covering various instruction formats like image captioning and visual question answering. The MLLM-DCL benchmark comprises multiple downstream tasks across distinct knowledge domains such as Remote Sensing, Medical, Autonomous Driving, Finance, and Science. Both benchmarks are chosen because the base model performs poorly on them in zero-shot settings, indicating a low risk of information leakage.
Performance is assessed using four integrated metrics: Mean Finetune Accuracy (MFT), which measures accuracy immediately after learning a task; Mean Final Accuracy (MFN), which reflects overall retained performance after all training; Mean Average Accuracy (MAA), providing a holistic view throughout the process; and Backward Transfer (BWT), quantifying the extent of forgetting.
Also Read:
- LiLoRA: A New Approach to Efficient Continual Learning in Multimodal AI
- MultiMedEdit: A New Benchmark for Updating Medical AI Knowledge
Key Findings and Future Directions
Experiments conducted with MCITlib reveal several insights. While MLLMs possess a strong inherent generalization ability that can partially mitigate catastrophic forgetting, a significant gap remains compared to individual task performance. Different methods show varying abilities to alleviate forgetting. Some, like ModalPrompt and HiDe, achieve lower forgetting but at the cost of learning new tasks less effectively, essentially “learning less and forgetting less.”
Among the evaluated methods, DISCO demonstrated the best overall performance on both benchmarks. This method extends parameters by storing specific LoRA parameters for each task, effectively mitigating forgetting but leading to considerable parameter overhead as the number of tasks grows. SEFE, a parameter regularization method, showed strong potential by preserving important parameters from previous tasks without adding extra parameters.
The research also highlighted that Multimodal Continual Instruction Tuning tasks present unique challenges due to their diverse modalities, task formats, and domain variations, leading to unpredictable or task-specific forgetting patterns. For instance, some tasks maintained stable performance, while others experienced significant degradation after learning just one new task.
MCITlib is designed to be continuously updated, incorporating the latest advancements in the MCL field. Future plans include expanding the library to support a greater diversity of base models, tasks, and evaluation dimensions, moving beyond just accuracy to consider factors like training/inference efficiency and the impact on the MLLM’s original generalization capabilities. The codebase is openly available for community contributions to help build a more comprehensive resource for Multimodal Continual Learning. You can find the research paper and codebase here: MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark.
