TLDR: This thesis presents a comprehensive suite of algorithms designed to enable deep neural networks to learn effectively from limited and imperfect real-world data. It addresses challenges in generative models (mode collapse, class confusion on long-tailed data), recognition models (overfitting tail classes, ViT limitations), semi-supervised learning (optimizing non-decomposable metrics), and domain adaptation (efficient sample selection and robust transfer learning). The research introduces novel techniques like NoisyTwins for GANs, SAM-based regularization for classifiers, Cost-Sensitive Self-Training and Selective Mixup for complex objectives, and S3VAADA/SDAT for efficient domain adaptation, significantly improving AI performance and practicality in diverse, challenging data environments.
In the rapidly evolving world of artificial intelligence, deep neural networks have achieved remarkable feats, from recognizing objects in images to generating realistic visuals. However, much of this success has relied on the availability of vast, meticulously organized, and balanced datasets. But what happens when the data isn’t perfect? What if it’s limited, skewed, or comes from a different environment than where the AI was trained?
This research delves into the critical challenge of enabling AI models to learn effectively from such “imperfect” real-world data. It addresses scenarios where some categories have abundant examples while others are scarce (known as long-tailed distributions), or where an AI trained in one setting needs to perform well in a completely new one (domain adaptation).
Crafting Smarter Generative AI
Generative AI, like the powerful Generative Adversarial Networks (GANs), can create stunningly realistic images. Yet, when trained on imbalanced datasets, these models often falter. They might either forget to generate images for rare categories entirely (a problem called “mode collapse”) or mix up different classes. This work introduces several clever solutions. One approach involves using a separate, pre-trained AI classifier to guide the GAN, ensuring it produces a balanced variety of images across all categories. Another technique, called Group Spectral Regularization (gSR), tackles a specific issue where internal parameters of the GAN “explode” for rare classes, leading to poor image quality. By keeping these parameters in check, gSR helps the GAN generate diverse and plausible images even for underrepresented categories.
For even more advanced generative models like StyleGANs, which are popular for their ability to create editable images, the challenge intensifies with large, diverse datasets. Here, the problem isn’t just mode collapse but also “class confusion,” where the AI generates an image of a car when it was asked for a truck. The research proposes “NoisyTwins,” a method that adds subtle “noise” to the AI’s internal representations and uses a self-supervised learning technique to ensure that images within the same category are diverse yet clearly distinct from other categories. This allows StyleGANs to produce high-quality, consistent, and varied images across thousands of classes, even those with very few training examples.
Enhancing AI’s Understanding with Targeted Learning
Beyond generating data, the research also explores how to make AI recognition models more robust when faced with limited data. Traditional methods often “re-weight” the importance of rare examples during training, but this can sometimes lead to the AI getting stuck in suboptimal learning states, like “saddle points” in its internal learning landscape. The study reveals that for rare categories, the AI often converges to these saddle points, hindering its ability to generalize. A technique called Sharpness-Aware Minimization (SAM) is shown to be effective in helping the AI escape these traps, leading to significantly better performance on rare classes.
The work also addresses a new generation of AI models called Vision Transformers (ViTs). Unlike older Convolutional Neural Networks (CNNs), ViTs lack built-in “intuition” about images, making them data-hungry and prone to overfitting on rare data. The research introduces “DeiT-LT,” a method that teaches ViTs how to learn from CNNs. By having the ViT learn from a CNN using “out-of-distribution” (unusual or augmented) images, and by encouraging it to learn “low-rank” (more general) features, DeiT-LT enables ViTs to perform exceptionally well on long-tailed datasets, even when trained from scratch.
Optimizing for Real-World Goals
In practical applications, simply achieving high accuracy isn’t always enough. Sometimes, it’s crucial to ensure that the AI performs well on the worst-case scenario, like maximizing the minimum recall across all patient groups in a medical diagnosis system. These are called “non-decomposable objectives” because they can’t be measured by looking at individual predictions alone. This research extends “self-training” methods, where AI models learn from large amounts of unlabeled data, to tackle these complex goals. The “Cost-Sensitive Self-Training (CSST)” framework allows AI to prioritize learning from underperforming categories, ensuring a more balanced and fair outcome.
Furthermore, for situations where an AI model is already trained but needs to be adapted for a specific, complex objective, the research proposes “SelMix.” This fine-tuning technique intelligently “mixes up” data from different classes during training, based on which combinations are most likely to improve the desired metric. SelMix can optimize for a wide range of objectives, including those that are non-linear, making it a versatile tool for adapting pre-trained AI models to nuanced real-world requirements.
Also Read:
- Enhancing Medical Image Analysis: A Hybrid Approach to Optimize Transfer and Self-Supervised Learning
- Compressing Knowledge: A New Approach to Self-Supervised Dataset Distillation
Seamless Adaptation to New Environments
Finally, the research tackles the challenge of “domain adaptation,” where an AI trained in one environment (e.g., on synthetic images) needs to perform well in another (e.g., on real-world photos). Instead of retraining from scratch or relying solely on unlabeled data, the work explores how to efficiently use a small amount of newly labeled data from the target environment. The “S3VAADA” method intelligently selects the most informative samples to label, considering their uncertainty, diversity, and how representative they are of the new domain. This ensures that every labeled example provides maximum benefit.
The study also delves into the underlying optimization processes of domain adaptation. It reveals that making the AI’s core “task loss” (e.g., for classification) smoother during training leads to better adaptation, while surprisingly, making the “adversarial loss” smoother can be detrimental. Based on this insight, “Smooth Domain Adversarial Training (SDAT)” is introduced, a technique that selectively smooths only the beneficial parts of the AI’s learning process. This leads to more stable and effective adaptation across different domains and tasks, even for advanced models like Vision Transformers. For more in-depth information, you can refer to the full thesis available at this link.
In essence, this comprehensive body of work provides a suite of innovative techniques that empower deep neural networks to thrive in the messy, imperfect data environments of the real world, pushing the boundaries of what AI can achieve in practical applications.
