TLDR: ELMM (Efficient Lightweight Multimodal Large Language Models) is a novel framework designed to improve Multimodal Knowledge Graph Completion (MKGC). It addresses challenges like semantic noise from excessive image tokens and high computational costs in MLLMs. ELMM introduces a Multi-view Visual Token Compressor (MVTC) to adaptively compress image tokens from both textual and visual perspectives, and an attention pruning strategy to remove redundant layers for faster inference. Experiments show ELMM achieves state-of-the-art performance and significantly boosts computational efficiency on benchmark datasets.
Multimodal Knowledge Graphs (MKGs) are advanced systems that combine traditional knowledge graphs with visual and textual information, creating richer and more expressive representations of entities. Imagine a knowledge graph where not only facts like “Taylor Swift is the girlfriend of Joe Alwyn” are stored, but also images of Taylor Swift and Joe Alwyn, along with textual descriptions. This integration allows for a deeper understanding of real-world knowledge.
However, a significant challenge with existing MKGs is their incompleteness. They often have missing information, which limits their usefulness in various applications. This has led to a growing focus on Multimodal Knowledge Graph Completion (MKGC), a task aimed at filling in these missing pieces by leveraging multimodal data.
While Large Language Models (LLMs) have shown great promise in traditional knowledge graph completion, their application to the multimodal setting, especially with Multimodal Large Language Models (MLLMs), has been largely unexplored. The direct application of MLLMs to MKGC faces two main hurdles: first, the sheer volume of image tokens per entity can introduce semantic noise and conflicts between different data types; and second, processing such large inputs demands high computational power, making it impractical for many real-time or large-scale scenarios.
Introducing ELMM: A New Approach to Multimodal Knowledge Graph Completion
To tackle these challenges, researchers have proposed a novel framework called Efficient Lightweight Multimodal Large Language Models (ELMM). ELMM is designed to make MLLMs more effective and efficient for MKGC by addressing the issues of data redundancy and computational cost.
How ELMM Works: Key Innovations
ELMM introduces two primary innovations:
1. Multi-view Visual Token Compressor (MVTC): This component uses a multi-head attention mechanism to intelligently compress image tokens. Instead of processing all image data, MVTC adaptively selects and compresses the most relevant image tokens from both textual and visual perspectives. This means it considers what parts of an image are important based on the accompanying text (like entity and relation descriptions) and also what visual information is intrinsically significant. This process significantly reduces visual noise, prevents modality conflicts, and ensures that only high-information tokens are retained, leading to better inference efficiency and stronger model representations.
2. Attention Pruning Strategy: MLLMs, like other large transformer-based models, often have redundant attention layers. ELMM identifies these layers by measuring the similarity between their input and output. Layers that show high similarity contribute little new information and can be pruned (removed) to reduce computational overhead during inference. To compensate for any performance degradation caused by this pruning, ELMM incorporates a linear projection mechanism with a carefully designed initialization strategy. This ensures that efficiency gains don’t come at a significant cost to accuracy.
Additionally, ELMM replaces the standard output layer of MLLMs with a specialized Multimodal Knowledge Reasoning Completion Layer. This layer is specifically designed to better fuse cross-modal information and generate accurate probability distributions for candidate entities, thereby improving the prediction of missing information in MKGs.
Also Read:
- GraphVista: A New Approach for Understanding Large-Scale Graphs with AI
- Improving Multimodal AI: Understanding How Modalities Work Together
Performance and Efficiency
Extensive experiments conducted on benchmark datasets, FB15k-237-IMG and WN18-IMG, demonstrate ELMM’s superior performance. It consistently outperforms existing state-of-the-art methods in MKGC, achieving significant improvements in accuracy metrics like Hits@1, Hits@3, and Hits@10. For instance, on the FB15k-237-IMG dataset, ELMM showed substantial gains over powerful multimodal approaches, with relative improvements of 26.8% in Hits@1, 18.8% in Hits@3, and 10.2% in Hits@10.
Beyond accuracy, ELMM also delivers substantial computational efficiency. It achieves over 30% faster inference times compared to non-pruned models, even when compared to some uni-modal (text-only) large language models. This efficiency, combined with its enhanced performance, establishes ELMM as a new standard for multimodal knowledge graph completion.
The research highlights the powerful potential of MLLMs when tailored for specific tasks like MKGC, and how strategic compression and pruning can unlock their capabilities without incurring prohibitive computational costs. For more details, you can read the full research paper here.
