TLDR: VISA is a novel method that enhances the inference efficiency of Multimodal Large Language Models (MLLMs) by intelligently selecting and aggregating visual tokens. It uses a graph-based approach to summarize visual information, preserving crucial details while significantly reducing the number of tokens, leading to faster processing without substantial performance loss. This is achieved through a Visual Token Aggregation (VTA) module and a Group-wise Token Selection Strategy (GTS), which together ensure a superior trade-off between model performance and inference speed across various MLLM architectures and tasks.
Multimodal Large Language Models (MLLMs) have made remarkable strides in understanding and interacting with both text and visual information. From answering questions about images to generating descriptions, these models are becoming increasingly sophisticated. However, their power comes with a significant computational cost, primarily due to the vast number of visual tokens they process. When an MLLM takes an image or video as input, it converts it into hundreds or even thousands of these visual tokens. This abundance of tokens can lead to slow inference speeds, especially with high-resolution images or video, hindering their real-world application.
Previous attempts to address this efficiency problem often involved simply pruning, or discarding, less important visual tokens. While this speeds up processing, it frequently results in a loss of crucial visual information, leading to a noticeable drop in the model’s performance. Other methods tried merging tokens, but often did so by simply averaging features, which can create a coarse-grained representation unsuitable for tasks requiring fine-grained visual details, like visual question answering.
Introducing VISA: A Smarter Approach to Visual Token Management
A new research paper introduces a novel method called group-wise VIsual token Selection and Aggregation (VISA) that aims to overcome these limitations. VISA offers a more sophisticated way to compress visual tokens, preserving more visual information than previous methods while significantly boosting inference efficiency. The core idea is to intelligently select and aggregate visual tokens, rather than just discarding them.
VISA is built upon two main components: the Visual Token Aggregation (VTA) module and the Group-wise Token Selection Strategy (GTS).
Visual Token Aggregation (VTA)
Imagine each visual token as a node in a network. The VTA module constructs a graph where connections between these nodes are based on their semantic similarity. If two tokens are semantically similar, they are connected. Instead of simply throwing away less important tokens, VTA aggregates their information into the more critical, ‘kept’ tokens (which can be thought of as ‘super nodes’) based on this graph network. This process, inspired by graph summarization techniques, creates a more compact representation of the visual input. It’s like summarizing a detailed report into a shorter, yet still comprehensive, executive summary – you keep the essential points and integrate the less critical details into them.
A key difference from simpler merging methods is that VTA allows information from a removed token to be propagated to multiple kept tokens, ensuring that no valuable detail is entirely lost but rather intelligently redistributed and summarized.
Group-wise Token Selection Strategy (GTS)
To ensure stable and accurate visual information extraction, VISA doesn’t just apply VTA once. Instead, it divides the layers of the MLLM into multiple groups. VTA is then applied progressively at the end of each group. This allows the model to process and refine the visual information in stages, ensuring that the ‘kept’ tokens steadily assimilate information from the ‘removed’ ones before the next round of pruning and aggregation.
The GTS also employs a smart way to decide which tokens to keep and which to remove. It uses the attention scores from the final text token to the visual tokens across multiple layers within each group. This means the model’s understanding of the question prompt guides the selection process, ensuring that visual information relevant to the query is prioritized and retained.
Also Read:
- Enhancing Multi-Image Question Answering in AI Models with Adaptive Visual Anchoring
- Unlocking Speed in Video LLMs: Verifier-Guided Token Pruning for Faster Decoding
Impressive Results Across Various Models
The researchers conducted extensive experiments on popular MLLMs like LLaVA-1.5 (7B and 13B parameters), LLaVA-NeXT (for high-resolution images), and Video-LLaVA (for video understanding). VISA consistently outperformed existing methods, achieving a superior balance between model performance and inference speed.
For instance, on LLaVA-1.5-13B, VISA was able to speed up inference by 2.08 times while retaining an impressive 98.14% of the model’s original performance. It also demonstrated strong performance in scenarios with high-resolution image inputs and video understanding tasks, where the token count is typically very high. This robust performance across different model architectures and input types highlights the versatility and effectiveness of the VISA method.
By intelligently selecting and aggregating visual tokens through graph summarization and a progressive, group-wise strategy, VISA offers a significant leap forward in making MLLMs more efficient without compromising their powerful understanding capabilities. This advancement paves the way for faster, more practical multimodal AI applications. You can find the full research paper here: VISA: Group-wise Visual Token Selection and Aggregation via Graph Summarization for Efficient MLLMs Inference.
