TLDR: FastDINOv2 introduces a two-stage frequency-based curriculum learning strategy for DINOv2, significantly reducing pre-training time and computational costs (1.6x faster, 2.25x fewer FLOPs) while maintaining or improving robustness to common image corruptions and competitive performance on various vision tasks. It achieves this by starting with low-frequency image features and then transitioning to full-resolution images with Gaussian noise patching to balance frequency biases.
Large-scale vision models like DINOv2 have shown impressive capabilities, but their training demands immense computational resources, making them difficult to reproduce or adapt for many researchers and organizations. This challenge limits further innovation and the application of these powerful models in various scenarios, such as with private datasets or new types of data.
A new research paper introduces FastDINOv2, a novel pre-training strategy for DINOv2 that aims to overcome these limitations. The core idea is to make the training process significantly faster while simultaneously improving the model’s resilience to common image corruptions, such as blur, noise, or changes in brightness.
The FastDINOv2 approach employs a unique two-stage curriculum learning strategy. In the first stage, the model is initially trained using only low-frequency features of images. This is achieved by downsampling the images, which helps the model quickly grasp broad, coarse patterns and accelerates its initial learning. For the first 75% of the training epochs, the model focuses on these simplified inputs.
In the second stage, which covers the remaining 25% of the training epochs, the model transitions to full-resolution images. Crucially, this stage also incorporates a new data augmentation technique called Gaussian noise patching. This involves replacing random patches within the images with Gaussian noise. This augmentation forces the model to learn to ignore high-frequency disturbances, thereby enhancing its robustness to various types of noise and fine-grained corruptions.
The combination of these two stages offers a dual benefit. By starting with low-frequency information, FastDINOv2 significantly speeds up the convergence of the training process. For instance, when applied to a ViT-B/16 backbone trained on ImageNet-1K, the pre-training time was reduced by 1.6 times, and the computational operations (FLOPs) were cut by an impressive 2.25 times compared to the standard DINOv2. Despite these efficiency gains, FastDINOv2 maintains competitive performance in standard image classification tasks and achieves comparable or even better robustness on corruption benchmarks like ImageNet-C.
The research highlights that robustness doesn’t necessarily require training at extreme scales, but can be effectively built into self-supervised learning models through thoughtful curriculum design and data augmentation. This makes advanced self-supervised foundation modeling more accessible and opens new avenues for exploring how data presentation and augmentation can improve model resilience.
Beyond efficiency and robustness, FastDINOv2 also demonstrates strong performance across various downstream tasks. It shows faster convergence in linear probing accuracy, improves instance-level recognition on datasets like Oxford and Paris, and maintains pixel-level understanding crucial for semantic segmentation tasks on ADE20K. Furthermore, the initial low-resolution training phase drastically reduces GPU memory consumption, making it feasible to train these large models on hardware with lower memory capacities for a significant portion of the training.
Also Read:
- Enhancing Model Robustness with Cross-Task Alignment in Test-Time Training
- Unlocking Data Groupings with Diffusion Models: Introducing CLUDI
This work represents a significant step towards making powerful vision foundation models more practical and widely usable, fostering further research and application in the field of computer vision. For more technical details, you can refer to the full research paper here.
