TLDR: This research paper introduces a novel method for fine-tuning CLIP models using a Bayesian approximation of Natural Gradient Descent via a Kalman filter. This approach addresses the challenges of few-shot learning by improving both in-distribution performance and out-of-distribution robustness, while also providing uncertainty quantification. The Kalman-based algorithm consistently achieves superior or comparable results against state-of-the-art baselines across various image classification datasets.
Vision-language models like CLIP have set new standards in how we process and understand multimodal data, combining both images and text. However, getting these powerful models to perform optimally on new, specific tasks, especially when only a small amount of labeled data is available, remains a significant challenge. This is particularly true for ensuring they work well not just on data similar to what they were trained on (in-distribution or ID) but also on new, unfamiliar data (out-of-distribution or OOD).
Most current methods for fine-tuning these models rely on basic optimization techniques that can be slow, sensitive to specific settings, and often struggle with OOD data. These methods typically use only the ‘first-order’ gradient information, which essentially tells them the steepest direction to go down in the model’s error landscape. But this landscape can be complex, with sharp turns and valleys, making these simple methods less effective.
A Smarter Approach to Fine-Tuning
A new research paper, titled “Bayesian Natural Gradient Fine-Tuning of CLIP Models via Kalman Filtering,” introduces a sophisticated solution to these problems. Authored by Hossein Abdi, Mingfei Sun, and Wei Pan from The University of Manchester, the paper proposes a novel method that combines the benefits of ‘second-order’ optimization with Bayesian inference. Second-order methods use more detailed information about the shape of the error landscape, allowing for more efficient and substantial updates per iteration, which is crucial when data is limited.
The core of their approach is a Bayesian approximation of Natural Gradient Descent (NGD) using a Kalman filter. NGD is a powerful second-order optimization technique that adjusts updates based on the local curvature of the loss function. While NGD is typically computationally intensive for large models, the researchers found a way to make it practical for CLIP models by integrating it with a Kalman filter.
Why Kalman Filtering and Bayesian Inference?
The Kalman filter, traditionally used for state estimation in dynamic systems, acts as a second-order optimizer within a Bayesian framework. This means it not only helps the model learn more efficiently but also provides ‘uncertainty quantification.’ This ability to understand how confident the model is in its predictions is key to improving its robustness and generalization to OOD data.
The researchers developed a ‘Kalman-based adapter’ to fine-tune CLIP models. This adapter allows the model to approximate the natural gradient direction, leading to better ID performance, while the Bayesian formulation inherently enhances OOD generalization by accounting for uncertainty. To further boost OOD robustness, the method dynamically adjusts its update steps based on how much new data deviates from the training distribution, using a measure called Mahalanobis distance.
Also Read:
- Visual Prompts for Balancing Safety and Responsiveness in Multimodal AI
- Preventing AI Forgetting: A New Routing Approach for Vision-Language Models
Demonstrated Superior Performance
Extensive experiments were conducted on various image classification datasets, including ImageNet, OxfordPets, Food101, SUN397, DTD, and EuroSAT for in-distribution scenarios, and distribution-shifted versions of ImageNet (ImageNetV2, ImageNet-Sketch, ImageNet-A, ImageNet-R) for out-of-distribution scenarios. The results consistently showed that their Kalman-based algorithm achieved superior or comparable ID performance and significantly improved OOD robustness compared to existing state-of-the-art methods like CoOp, CLIP-Adapter, and Tip-Adapter-F.
For instance, on datasets like OxfordPets, Food101, and SUN397, the algorithm showed notable performance gains, especially with more labeled examples. In OOD tests, it achieved the highest average accuracy across distribution-shifted datasets. The study also explored how different settings (like the ‘scaling factor’ and ‘forgetting factor’) influenced the model’s robustness, demonstrating that careful adjustment can lead to even better performance, particularly when dealing with corrupted or out-of-distribution data during training.
This work marks the first successful application of Kalman filtering to fine-tune CLIP-based models, paving the way for more robust and efficient learning in vision-language tasks. For more details, you can read the full research paper here.
