TLDR: MobileCLIP2 introduces a new family of image-text models that significantly improve upon previous MobileCLIP models by enhancing multi-modal reinforced training. This is achieved through better CLIP teacher ensembles and improved captioner teachers, both leveraging the DFN dataset and fine-tuning on high-quality image-caption data. The new models, including MobileCLIP2-S4, achieve state-of-the-art ImageNet-1k zero-shot accuracies at low latencies, often being smaller and faster than comparable models. The research also introduces new architectures optimized for higher resolutions and releases pretrained models and data generation code to foster wider adoption and development.
Foundation models like CLIP have transformed how we approach image and text understanding, offering impressive zero-shot capabilities that allow them to perform tasks without explicit prior training. These models, however, often come with a significant computational cost, making them challenging to deploy on devices with limited resources, such as mobile phones.
This is where MobileCLIP, a family of image-text models, previously made strides by focusing on low-latency and light architectures. It introduced a novel multi-modal reinforced training method that efficiently distilled knowledge from multiple caption generators and CLIP teachers. Now, a new paper, MobileCLIP2: Improving Multi-Modal Reinforced Training, takes these advancements even further, presenting a new generation of models that push the boundaries of efficiency and accuracy.
Enhancing Multi-Modal Reinforced Training
The core of MobileCLIP2’s improvements lies in refining the multi-modal reinforced training process. The researchers focused on two key areas: better CLIP teacher ensembles and improved captioner teachers. They achieved this by:
- Utilizing superior CLIP teacher ensembles trained on the DFN dataset.
- Developing enhanced captioner teachers, also trained on the DFN dataset, and then fine-tuning them on a diverse selection of high-quality image-caption datasets.
Through extensive experiments, the team uncovered crucial insights, such as the importance of fine-tuning temperature in contrastive knowledge distillation, the effectiveness of caption-generator fine-tuning for creating more diverse captions, and the additive benefits of combining synthetic captions from multiple models.
Achieving State-of-the-Art Performance at Low Latencies
The result of these improvements is a new family of models called MobileCLIP2, which achieves state-of-the-art ImageNet-1k zero-shot accuracies while maintaining remarkably low latencies. For instance, MobileCLIP2-B shows a significant 2.2% improvement in ImageNet-1k accuracy compared to its predecessor, MobileCLIP-B.
Perhaps even more impressively, MobileCLIP2-S4 matches the zero-shot accuracy of SigLIP-SO400M/14 on ImageNet-1k, despite being twice as small. It also outperforms DFN ViT-L/14 with 2.5 times lower latency. These figures highlight MobileCLIP2’s ability to deliver high performance in a compact and fast package, making it ideal for mobile and edge device applications.
Under the Hood: Training Improvements
The paper details several key improvements to the training methodology:
- Better Base Dataset: MobileCLIP2 leverages the DFN-5B dataset, which is a higher-quality, filtered dataset, offering better performance compared to the DataComp-1B dataset used previously.
- DFN CLIP Teachers: The researchers investigated the effectiveness of DFN-pretrained models as CLIP teachers. They found that an ensemble of DFN2B-CLIP-ViT-L-14-s39b and DFN2B-CLIP-ViT-L-14 teachers, combined with optimal logit scaling, significantly boosted performance.
- DFN Caption Generators: A new CoCa model was pretrained on DFN-2B and then fine-tuned on high-quality datasets like MSCOCO-38k. This approach improved ImageNet-1k validation and average performance across 38 evaluations, while also recovering retrieval performance. The study also explored the impact of synthetic caption diversity, finding that while multiple models can generate diverse captions, the performance gains for downstream tasks were within a standard deviation.
New Architectures for Wider Applications
MobileCLIP2 introduces new architectural variants, MobileCLIP2-S3 and MobileCLIP2-S4, which feature a 5-stage design for their image encoders. This design allows for better distribution of parameters and more effective scaling to higher resolutions, which is crucial for tasks like image segmentation that require high input image resolutions.
Also Read:
- Unlocking Commonsense in Compact Vision-Language Models
- Bridging Vision and Language: The Evolution of Object Detection with Advanced AI Models
Broader Impact
By optimizing foundation models for mobile and edge devices, MobileCLIP2 facilitates the broader use of these powerful AI tools and enables the development of applications for a wider user base. The researchers have also released their pretrained models and data generation code, making it easier for others to create new reinforced datasets with arbitrary teachers using distributed scalable processing.
This work represents a significant step forward in making advanced multi-modal AI more accessible and efficient, paving the way for innovative applications on resource-constrained devices.
