TLDR: This research introduces a two-stage framework for robust classification using foundation models when training data contains noisy labels, without requiring model retraining. It leverages the local geometry of the model’s embedding space, primarily through the Non-Negative Kernel (NNK) algorithm, to estimate the reliability of individual data samples. Various reliability metrics, including NNK-based and k-means clustering approaches, are proposed. These reliability scores then guide a weighted majority voting inference process. Experiments on CIFAR-10 and DermaMNIST datasets demonstrate that geometry-aware methods improve classification accuracy across different noise conditions, outperforming standard baselines. The study concludes that local geometry is more effective at low noise levels, while global clustering methods are superior in high-noise scenarios.
Foundation Models (FMs) have become incredibly important in machine learning, especially because they are pre-trained on vast datasets and can be easily adapted for many different tasks. However, a significant challenge arises when these models need to be fine-tuned with data that contains incorrect or ‘noisy’ labels. Retraining these large models from scratch is often too expensive and time-consuming.
Current research on handling noisy labels generally falls into three categories: selecting clean data samples, adjusting the loss functions used during training, or leveraging the model’s embedding space. While sample selection and loss adjustment methods have their merits, they can be computationally intensive or suffer from issues like overfitting and limited interpretability.
This new research, detailed in the paper Robust Classification under Noisy Labels: A Geometry-Aware Reliability Framework for Foundation Models, focuses on the embedding space approach. The embedding space is essentially a high-dimensional representation of the data generated by the foundation model. The core idea is to use the ‘local geometry’ within this space to determine how reliable each training data sample is, without needing to retrain the entire model.
The paper introduces a two-stage framework. The first stage involves estimating the ‘reliability’ of each training data sample. The second stage uses these reliability scores to make more robust classification decisions for new, unseen data. This approach builds upon previous work, like the Weighted Adaptive Nearest Neighbor (WANN) method, but addresses its limitations by incorporating more sophisticated geometric information.
Estimating Reliability with Geometry
A key innovation in this framework is the use of the Non-Negative Kernel (NNK) algorithm. Unlike traditional k-nearest neighbor (k-NN) methods that simply pick the ‘k’ closest points, NNK constructs a more refined local neighborhood. It identifies a subset of neighbors that are geometrically non-redundant and assigns weights based on their similarity. This allows for a more nuanced understanding of the local data structure.
The researchers propose several ways to estimate reliability:
-
k-NN Reliability: A basic measure, where reliability is the fraction of k-nearest neighbors sharing the same label as the sample. This doesn’t account for geometry.
-
NNK Weights Reliability: This method uses the NNK algorithm’s assigned weights. If a sample’s neighbors with the same label have high NNK weights, it’s considered more reliable.
-
NNK Diameter Ratio Reliability: This is a more geometry-aware metric. It compares the ‘diameter’ (maximum distance between points) of a sample’s overall NNK neighborhood to the diameter of its neighborhood consisting only of same-class neighbors. A smaller ratio indicates higher reliability.
-
Supervised k-means Reliability: For scenarios with higher noise, local neighborhoods can become less trustworthy. This method uses global clustering. It runs k-means clustering for each class and calculates reliability based on how close a sample is to the centroids of its own class.
-
Unsupervised k-means Soft Clustering Reliability: This is another global approach, but it determines cluster centers purely based on geometric principles, without using labels initially. Each cluster then gets a ‘soft’ label distribution based on the classes of samples within it. Reliability is then based on how well a sample aligns with its nearest cluster’s probabilistic label distribution.
Making Predictions with Reliability
Once reliability scores are estimated, they are used in the inference stage to classify new data. This is done through a ‘weighted majority voting’ approach. Each neighbor’s vote is adjusted by its reliability score. The paper explores two modes: ‘weighted’ mode, where reliability is multiplied by the NNK edge weight, and ‘unweighted’ mode, where only reliability is considered.
Experimental Validation
The framework was tested on two standard vision datasets: CIFAR-10 and DermaMNIST (a more challenging medical dataset). Images were processed using a pre-trained DINOv2 model to obtain embeddings. The experiments involved injecting both symmetric (random label switching) and asymmetric (specific label changes) noise at various levels.
The results showed that the proposed methods generally improved robustness compared to standard k-NN, ANN, and WANN baselines. Key findings include:
-
Local vs. Global: Local geometry-based methods (like NNK diameter reliability) performed best with clean or low-noise data, as precise distances are highly informative. As noise increased, global clustering methods (like k-means) became more effective because cluster centers are more stable against label errors.
-
Weighted vs. Unweighted Inference: At lower noise levels, weighted majority voting (using both NNK weights and reliability) was better. However, at higher noise levels, unweighted voting (relying solely on reliability) performed better, suggesting that distances in local neighborhoods become less reliable when many labels are wrong.
-
Dataset Complexity: For simpler datasets like CIFAR-10, local geometry was very useful. For more complex datasets like DermaMNIST, where the embeddings themselves might be ‘noisy,’ unsupervised k-means and unweighted NNK weights reliability performed best at higher noise levels.
-
Supervised vs. Unsupervised K-means: While supervised k-means performed well in low-noise asymmetric scenarios, unsupervised k-means proved more resilient to heavier noise, especially for the more complex DermaMNIST dataset. This indicates that geometry-only clustering is more beneficial when label errors make per-label clusters unreliable.
Also Read:
- Improving Graph Learning Across Domains with Noisy Labels
- Enhancing Trustworthiness in Language Models: A Deep Dive into Calibration and Label Smoothing
Conclusion
This research highlights the power of geometry-aware reliability estimation for robust classification with foundation models, particularly in the presence of noisy labels, without the need for expensive model retraining. The findings suggest that the optimal approach depends on the level and type of label noise, with local geometry being crucial for low noise and global clustering for high noise. Future work aims to develop adaptive hybrid methods that can dynamically balance these approaches.
