TLDR: A new method called Semantic Visual Projector (SVP) significantly improves the efficiency of Multimodal Large Language Models (MLLMs) for Referring Image Segmentation (RIS). It achieves this by using SAM-generated “semantic superpixels” as visual tokens, reducing the number of tokens by about 93% without losing performance. This approach, combined with Semantic Superpixel Positional Embedding (SSPE) and a Semantic Superpixel Aggregator (SSA), speeds up MLLM training and inference by 60-70% and outperforms other visual compression techniques.
Referring Image Segmentation (RIS) is a crucial task in computer vision, enabling users to precisely locate and segment objects within an image using natural language descriptions. This capability is highly valuable for applications like image editing and robotic perception, as it allows for flexible object identification beyond predefined categories. Recent advancements have seen Multimodal Large Language Models (MLLMs) paired with powerful segmentation models like the Segment Anything Model (SAM) achieve impressive results in RIS.
However, a significant challenge in adapting MLLMs for segmentation is computational intensity, primarily due to the large number of visual tokens processed. Traditional methods, which often divide images into fixed-size patches, face a dilemma: using large patches can reduce token count but risks blending different semantic regions, leading to ambiguity. Conversely, small patches preserve detail but create a redundant and computationally expensive sequence of visual tokens. This trade-off often forces models to retain excessively long token sequences to maintain performance.
Inspired by how text tokenizers break down language into meaningful subwords, researchers have proposed a novel solution: the Semantic Visual Projector (SVP). This innovative approach leverages semantic superpixels generated by SAM to identify “visual words” within an image. By compressing and projecting these semantic superpixels as visual tokens, the SVP adaptively shortens the token sequence based on the complexity of the scene, minimizing semantic loss during compression. For a deeper dive into the technical details, you can read the full research paper here.
Enhancing Spatial Awareness and Context
To further enhance the MLLM’s understanding of image structure, two key components are introduced: the Semantic Superpixel Positional Embedding (SSPE) and the Semantic Superpixel Aggregator (SSA).
Traditional positional embeddings, designed for uniform patches, struggle to represent the varied shapes, sizes, and relative positions of semantic superpixels. The SSPE addresses this by adaptively encoding the geometry and position of each superpixel using multiple learnable queries. This enriched positional information is then injected into the visual tokens, strengthening the MLLM’s awareness of the superpixels’ spatial relationships.
The SSA plays a crucial role in preserving both fine-grained details within superpixels and the broader global context. It refines the semantic superpixel embeddings by intelligently aggregating image information. By sharing SSPE at the superpixel level in cross-attention layers, the SSA encourages the model to prioritize details within its corresponding superpixel while also integrating relevant surrounding environmental context. This mechanism helps the MLLM better distinguish target objects, especially when dealing with similar objects or complex relational descriptions.
Also Read:
- Boosting Vision Transformer Efficiency for Semantic Segmentation with Dynamic Token Reduction
- Boosting Efficiency in UI Code Generation with Smart Token Compression
Significant Efficiency Gains and Performance
Experiments demonstrate that this new method dramatically cuts visual tokens by approximately 93% on average without compromising performance. This leads to a notable acceleration in MLLM training and inference times, reducing them by 60% to 70%. The SVP also significantly outperforms existing compressive visual projectors on RIS tasks. The research also highlights the importance of using a vision encoder like DINOv2, which captures fine-grained semantics across all image positions, as opposed to CLIP-L, which can overlook attribute information in non-salient regions.
The core idea behind this efficiency boost is that the MLLM’s computational complexity grows quadratically with input token length. By drastically reducing the number of visual tokens, the SVP alleviates the primary bottleneck in MLLM-based RIS frameworks, which is often the Large Language Model itself. While the SVP introduces a minimal overhead, the computational savings it provides to the LLM far outweigh this cost, leading to a substantial improvement in overall efficiency.
This research, conducted by Xiaobo Yang and Xiaojin Gong from Zhejiang University, China, marks a significant step towards making powerful MLLMs more practical and efficient for complex pixel-level tasks like Referring Image Segmentation, paving the way for broader applications in various multimodal scenarios.
