TLDR: GranViT is a new Vision Transformer that improves Multi-modal Large Language Models (MLLMs) by focusing on fine-grained visual details. It uses a large dataset called Gran-29M with millions of region-level annotations and a two-stage training process. This process involves training the vision encoder to extract detailed features (Bbox2Caption task with self-distillation) and then adapting the LLM to utilize these features for precise localization (Caption2Bbox task). GranViT achieves state-of-the-art results in fine-grained recognition, OCR, and multimodal VQA, demonstrating strong transferability to various LLMs.
Multi-modal Large Language Models (MLLMs) have made significant strides in understanding both text and images. However, a common limitation among existing vision encoders in these models is their focus on general, global image features, often overlooking the crucial fine-grained details necessary for truly nuanced visual understanding. This gap limits their performance in tasks requiring precise regional analysis, such as identifying specific objects or reading text within an image.
Addressing this challenge, researchers have introduced GranViT, a new Vision Transformer designed to enhance MLLMs with a much finer-grained perception. GranViT integrates detailed feature extraction with semantic alignment to Large Language Models (LLMs) through a unique region-level autoregressive training approach.
The Gran-29M Dataset: A Foundation for Fine-Grained Learning
A key innovation behind GranViT is the creation of Gran-29M, a massive dataset comprising 29 million natural and OCR (Optical Character Recognition) images. What makes this dataset special is its extensive collection of over 180 million high-quality region-level annotations. These annotations go beyond simple image labels, providing specific details about objects and text within defined bounding boxes. This rich, fine-grained data is essential for training models to understand local visual information.
To build Gran-29M, the team leveraged existing datasets like UMG-41M, FLICKR30k, and LAION for natural images, generating both image-level and region-level annotations using tools like ViTDet and Qwen2.5-VL. For OCR images, publicly available datasets were consolidated, and PaddleOCR was used to provide precise bounding boxes and textual content for localized text. A rigorous filtering process ensured the high quality of these annotations, focusing on aspects like bounding box aspect ratio, area, quantity, and image resolution.
A Novel Pretraining-Adaptation Framework
GranViT employs a two-stage pretraining-adaptation framework to effectively learn from this fine-grained data:
- Stage 1: Fine-Grained Feature Extraction (Vision Encoder Tuning)
In the first stage, the vision encoder and a projection module are trained, while the LLM remains frozen. This stage focuses on enhancing the vision encoder’s ability to extract fine-grained features. It uses a “bounding-box-to-caption” (Bbox2Caption) task, where the MLLM is prompted with bounding box coordinates and asked to generate a localized caption for the content within that box. This direct supervision on local features significantly improves the visual representations. Additionally, a self-distillation mechanism is incorporated, where a “teacher” vision encoder guides the “student” vision encoder to further strengthen its regional reasoning capabilities by imposing explicit localization constraints.
- Stage 2: LLM Adaptation and Transfer (Projector and LLM Tuning)
The second stage focuses on adapting the pretrained vision encoder to various LLMs and strengthening the LLM’s localization capabilities. Here, the vision encoder is frozen, and the projector and LLM are tuned. This stage utilizes a “caption-to-bounding-box” (Caption2Bbox) task, where the MLLM is given a caption and asked to output the bounding box coordinates of the described region. This ensures that the LLM can effectively utilize the fine-grained visual features provided by the vision encoder for accurate localization and transferability to different LLM architectures.
This progressive training paradigm, combined with autoregressive caption loss, ensures that the vision encoder develops enhanced fine-grained feature extraction abilities, with its outputs naturally aligned to the LLM’s semantic space. The LLM, in turn, becomes more adept at using visual information for precise localization, leading to better object recognition and spatial reasoning in downstream tasks and reducing errors like visual hallucinations.
Also Read:
- MedAlign: A New AI Framework for Accurate and Efficient Medical Imaging Analysis
- Chunk-GRPO: A New Approach to Text-to-Image Generation
Outstanding Performance Across Diverse Tasks
Extensive experiments demonstrate GranViT’s superior performance. It surpasses existing vision encoders, achieving state-of-the-art results in fine-grained recognition, multimodal Visual Question Answering (VQA), and OCR understanding. For instance, in fine-grained and OCR tasks, GranViT achieved average top-1 scores of 80.78 and 55.97, respectively, outperforming the next best models by notable margins. While it maintains comparable performance in multimodal VQA and reasoning tasks, its primary strength lies in its detailed visual perception.
GranViT also shows strong transferability, maintaining outstanding performance when adapted to larger LLMs like Qwen2.5-3B and Qwen2.5-7B, even though it was initially pretrained with a lighter LLM. This flexibility makes it a robust foundation for various MLLM applications.
In conclusion, GranViT represents a significant advancement in multimodal understanding by addressing the critical need for fine-grained visual perception in MLLMs. Through its innovative dataset and two-stage training framework, it equips MLLMs with the ability to not only see but also precisely understand the intricate details within images, paving the way for more accurate and reliable AI systems. For more details, you can refer to the research paper.
