TLDR: CLID-MU is a novel meta-learning strategy that enables deep neural networks to learn effectively from noisy labeled data without requiring a separate clean dataset. It achieves this by introducing Cross-Layer Information Divergence (CLID), an unsupervised metric that evaluates model performance based on the consistency of data structures across different network layers, guiding the training process and outperforming existing methods.
Training deep neural networks often requires vast amounts of accurately labeled data. However, in real-world scenarios, obtaining such pristine datasets is a significant challenge due to the high cost and difficulty of data labeling. This frequently leads to datasets with “noisy labels,” where some of the assigned categories are incorrect. Learning with noisy labels (LNL) is a critical area of research focused on enabling deep neural networks to perform robustly even with imperfect data.
Meta-learning approaches have shown great promise in tackling the problem of noisy labels. These methods typically rely on a small, clean, and unbiased dataset, known as a meta-dataset, to guide the training of the main model. This meta-dataset helps evaluate the model’s performance and steer the learning process effectively. However, the core limitation of this approach is its heavy dependence on the availability of such a clean meta-dataset, which is often just as difficult to acquire as the large, clean training data itself.
When a clean meta-dataset isn’t available, existing meta-learning methods face significant hurdles. Some try to use robust loss functions with noisy meta-datasets, but these struggle with complex noise patterns. Others attempt to select a “pseudo-clean” subset from the noisy data to act as the meta-dataset, but this often requires careful tuning and can still be unreliable, especially with instance-dependent noise where the noise varies for each data point. This unreliability can mislead the training process and cause the model to overfit to the noisy labels.
A new approach, called CLID-MU (Cross-Layer Information Divergence-based Meta Update Strategy), addresses this fundamental challenge by entirely bypassing the need for a clean labeled meta-dataset. Instead, CLID-MU leverages the data itself, without relying on any labels, to guide the meta-learning process. The core insight behind CLID-MU is that clean data samples tend to maintain a consistent structure between different layers of a deep neural network, specifically between the last hidden layer and the final output layer. Noisy samples, on the other hand, disrupt this consistency.
CLID-MU introduces an unsupervised metric called Cross-Layer Information Divergence (CLID). This metric quantifies the divergence, or difference, in the data distribution between the feature embeddings generated by the last hidden layer and the class probabilities produced by the final output layer. Essentially, it measures how well the internal representations of the data align with the model’s final predictions, all without needing to know the true labels. The researchers demonstrate that CLID closely correlates with the actual model performance, even in the absence of clean labels.
This crucial correlation allows CLID to serve as the meta-loss in the meta-training step. Unlike previous methods that use supervised loss functions (which depend on labels) as meta-objectives, CLID-MU uses its label-independent CLID metric. This means the meta-model, which guides the overall training, is updated based on an evaluation that is not susceptible to the noise in the labels. This significantly reduces the risk of overfitting to noisy labels, a common pitfall in other approaches.
The CLID-MU strategy works by dynamically measuring the CLID of the model for each training batch. This measurement provides valuable, label-independent guidance to the model training process. The meta-model is updated using meta-gradients derived from these CLID calculations, and in turn, the meta-model provides signals to enhance the performance of the main classification model. The paper also suggests using snapshot ensembling, where the best-performing models (selected based on their CLID scores) are combined during inference to further improve accuracy.
Extensive experiments on various benchmark datasets, including those with synthetic and real-world noise, demonstrate that CLID-MU consistently outperforms state-of-the-art methods. It shows superior performance in scenarios with high noise levels and complex noise patterns. Furthermore, CLID-MU proves to be robust in semi-supervised settings, where only a small portion of the data is noisily labeled, and the majority remains unlabeled. The method is also shown to be relatively insensitive to the tuning of its key hyperparameters, such as temperature scaling and batch size.
Also Read:
- Unlocking New Discoveries: A Multi-View Approach to Identifying Unseen Data Categories
- Enhancing Gaussian Processes with Neural Networks for Dynamic Data Patterns
CLID-MU represents a significant step forward in learning with noisy labels. Its ability to operate without a clean validation set makes it highly practical for real-world applications where clean data is scarce. The method is also compatible with other existing LNL techniques, suggesting it can be integrated to further enhance their performance. For more technical details, the full research paper can be accessed at https://arxiv.org/pdf/2507.11807.
