TLDR: MemEIC is a novel method for Continual and Compositional Knowledge Editing (CCKE) in large vision-language models (LVLMs). It addresses limitations of previous methods by enabling sequential and compositional editing of both visual and textual knowledge. MemEIC uses a hybrid external-internal editor with dual external memory for cross-modal evidence retrieval, dual LoRA adapters for disentangled parameter updates, and a brain-inspired knowledge connector for selective cross-modal integration. This approach significantly improves performance on complex multimodal questions and effectively preserves prior edits, setting a new benchmark for CCKE.
In the rapidly evolving world of artificial intelligence, keeping large vision-language models (LVLMs) up-to-date with new information is a constant challenge. These powerful AI systems, which combine both visual and textual understanding, often struggle when information changes or when they need to integrate new facts across different types of data. Traditional knowledge editing methods typically focus on updating either visual or textual information in isolation, overlooking the complex interplay between them and the continuous nature of real-world knowledge updates.
Addressing these critical limitations, a new research paper introduces MemEIC, a groundbreaking approach to what the researchers call Continual and Compositional Knowledge Editing (CCKE) in LVLMs. MemEIC is designed to allow these advanced AI models to learn and integrate new visual and textual facts sequentially and compositionally, meaning it can combine multiple updated pieces of information to answer complex questions.
How MemEIC Works: A Brain-Inspired Approach
MemEIC employs a clever hybrid system that combines both external and internal memory mechanisms, drawing inspiration from how the human brain processes and stores information. The system has several key components:
Query Decomposition: First, when a user asks a question, MemEIC automatically breaks it down into its visual and textual parts. For example, a question like “What position did the person in the photo recently assume?” would be split into “Who is the person in the photo?” (visual) and “What position did Donald Trump recently assume?” (textual, after identifying the person).
Modality-Aware External Memory (Mem-E): This part acts like a dual external notebook, storing visual and textual edits separately. When information is needed, it retrieves relevant facts using both image and text cues, ensuring no crucial detail is missed. This is a significant improvement over older methods that often relied only on text for retrieval, leading to poorer visual editing performance.
Internal Separated Knowledge Integration (Mem-I): Inspired by the brain’s lateralization (where different hemispheres specialize in different functions), MemEIC uses separate, lightweight “LoRA adapters” within the model’s internal structure for visual and textual knowledge. This separation is crucial because it prevents editing a visual fact from accidentally distorting textual information, and vice versa. This avoids a common problem in AI models called “representation collapse” or “catastrophic forgetting,” where new updates erase old, valuable knowledge.
Knowledge Connector: Perhaps the most innovative component, this is a “brain-inspired knowledge connector,” likened to the corpus callosum in the human brain. This connector selectively links the visual and textual pathways, fusing edited knowledge from both modalities only when a question requires combining information from both. If a question is purely visual or textual, the two streams remain separate. This selective connection is what enables MemEIC to perform robust compositional reasoning, even when there might be conflicts between external and internal knowledge.
Also Read:
- Why AI Agents Falter in User Interfaces: A Deep Dive into Missing GUI Knowledge
- The Internal Architecture Behind Text Bias in Multimodal Language Models
Setting a New Standard for AI Editing
To properly evaluate these capabilities, the researchers also introduced the Continual and Compositional Knowledge Editing Benchmark (CCKEB). This new benchmark is the first to assess AI models under continual knowledge editing (multiple sequential updates) and compositional queries that require integrating information from both visual and textual edits. It also includes a new metric called “Compositional Reliability” (CompRel) to measure how well a model can combine updated knowledge.
Experiments show that MemEIC significantly outperforms existing knowledge editing methods. It achieves much higher performance on complex multimodal questions, effectively preserves prior edits over time, and demonstrates strong robustness against catastrophic forgetting. The Knowledge Connector, in particular, proved vital, boosting compositional reliability by a substantial margin compared to other approaches.
This research marks a significant step forward in making large vision-language models more adaptable and reliable in dynamic, real-world scenarios, from correcting misidentified individuals in images to updating factual statements about them. The project is available for further exploration at https://github.com/MemEIC/MemEIC.
The paper, titled “MemEIC: A Step Toward Continual and Compositional Knowledge Editing,” was authored by Jin Seong, Jiyun Park, Wencke Liermann, Hongseok Choi, Yoonji Nam, Hyun Kim, Soojong Lim, and Namhoon Lee.
