TLDR: MVP-LM is a new framework that unifies various computer vision perception tasks, including object detection, segmentation, and visual grounding, under a single Visual Large Language Model (VLLM). It addresses the limitation of existing models that typically specialize in only a subset of tasks by integrating both word-based and sentence-based instructions with box and mask predictions. Key innovations include a multi-granularity decoder and a Chain-of-Thought-inspired data unification strategy, enabling versatile and accurate perception across diverse benchmarks.
In the evolving landscape of artificial intelligence, a new framework named MVP-LM is making strides in computer vision. This innovative system, developed by researchers from Tsinghua Shenzhen International Graduate School and Meituan Inc., aims to unify various perception tasks within a single Visual Large Language Model (VLLM) framework. Traditionally, computer vision models have specialized in a limited subset of perception tasks, which often restricts their real-world applicability.
Perception tasks in computer vision can be broadly categorized into four groups based on two key dimensions: the type of prediction (bounding box or segmentation mask) and the type of instruction (word-based or sentence-based). For instance, identifying an object by a single word like “car” is word-based, while locating “the red car parked near the tree” is sentence-based. Similarly, predicting a simple box around an object differs from generating a detailed pixel-level mask of its shape. Existing models often excel in one or two of these combinations but struggle to cover all four comprehensively.
MVP-LM, which stands for Multi-granular and Versatile Perception framework, addresses this challenge by integrating both word-based and sentence-based perception tasks, along with both box and mask predictions, into one cohesive architecture. This means the model can handle a wide range of tasks, including panoptic segmentation (identifying all objects and their pixel-level masks), object detection (locating objects with bounding boxes), visual grounding (finding objects based on textual descriptions), and referring expression segmentation (segmenting objects described by a sentence).
How MVP-LM Works
The framework incorporates an innovative multi-granularity decoder and a unique data unification strategy inspired by Chain-of-Thought (CoT) prompting. The architecture consists of four main components: an image encoder, a connector module, a Large Language Model (LLM), and the multi-granularity decoder. The image encoder processes visual information, the connector bridges the visual and textual data, and the LLM helps in understanding user queries and generating responses. The multi-granularity decoder is crucial for producing both bounding boxes and detailed segmentation masks simultaneously.
A key innovation is the dynamic query generation process. MVP-LM uses the VLLM to create a special “summary token” that captures essential information from the input. This summary is then used to generate “base query vectors.” At the same time, the model calculates the similarity between visual features from the image and the text instructions. The most relevant visual features are then combined with the base query vectors to create refined queries. This dynamic process allows MVP-LM to focus on specific regions of an image, leading to more accurate predictions.
The CoT-inspired dataset unification strategy is another significant aspect. The researchers transformed various datasets, which originally had different types of annotations (like boxes or masks), into a unified format. This involved creating question-answer pairs for each data sample, encouraging the model to adopt a “thinking-then-perceiving” approach. To further enhance the model’s understanding, a multi-caption auto-labeling technique was employed, where open-source VLLMs generate multiple descriptive captions for images lacking them, enriching the training data.
Also Read:
- Unlocking Robust Video Object Segmentation with Concept-Driven AI
- ChartScope: Advancing AI’s Understanding of Visual Data
Performance and Future Outlook
Extensive experiments have shown that MVP-LM delivers competitive performance across various benchmarks for both word-based and sentence-based perception tasks. It has demonstrated strong results in closed-set tasks (where categories are predefined) and exceptional performance in open-set segmentation (where the model encounters new categories). For instance, on the open-set ADE20K dataset, MVP-LM showed substantial improvements over previous VLLM-based methods.
The paper highlights MVP-LM’s ability to achieve state-of-the-art performance in referring expression segmentation (RES) and competitive results in referring expression comprehension (REC), often outperforming models with significantly more parameters. This indicates the robustness and generalization capabilities of the framework.
In conclusion, MVP-LM represents a significant step towards a unified framework for visual perception, capable of handling diverse tasks with varying granularities and instruction types. The researchers plan to further explore the application of reinforcement learning-like training in perception, building on the success of their CoT-inspired data curation. For more technical details, you can read the full research paper here.
