TLDR: SAMOSA is a new Open-Set Active Learning (OSAL) algorithm that improves machine learning model accuracy by intelligently selecting “atypical” yet highly informative data samples for labeling. It uses a novel metric called SAMIS-P, derived from the performance difference between Sharpness-Aware Minimization (SAM) and Stochastic Gradient Descent (SGD) models, to identify these crucial samples. Experiments show SAMOSA outperforms state-of-the-art methods by up to 3% accuracy, especially in challenging scenarios with many unknown classes, and is robust to label noise.
Modern machine learning systems often require vast amounts of labeled data, which can be expensive and time-consuming to acquire. To alleviate this burden, a field called Active Learning (AL) focuses on intelligently selecting the most informative samples from a large pool of unlabeled data for human annotation. However, real-world scenarios introduce a significant challenge: the unlabeled data might contain “irrelevant” or “unknown” classes, a problem known as Open-Set Active Learning (OSAL).
Traditional active learning methods often struggle in OSAL settings because they might waste resources labeling samples that don’t belong to the target classes. The key challenge in OSAL is to effectively filter out these irrelevant samples while simultaneously identifying the most valuable, informative samples from the known classes.
A new research paper introduces an innovative solution called Sharpness-Aware Minimization for Open-Set Active Learning, or SAMOSA. This algorithm offers a highly effective way to query samples in OSAL. SAMOSA builds on recent theoretical insights about how data “typicality” influences the generalization abilities of standard optimization methods like Stochastic Gradient Descent (SGD) and a more robust variant called Sharpness-Aware Minimization (SAM).
At its core, SAMOSA actively selects samples based on their “typicality.” It excels at identifying atypical samples—those that are unusual but highly informative. These samples often reside in regions of the data’s embedding space that are close to the model’s decision boundaries. By prioritizing these atypical samples, SAMOSA ensures that the selected data points are both highly informative for the target classes and useful for distinguishing between target and unwanted classes.
The methodology behind SAMOSA involves a clever metric called SAMIS-P (SAM mInus Sgd-Probabilities). This metric measures the difference in output prediction probabilities between a SAM-trained model and an SGD-trained model. The theoretical foundation suggests that a larger difference in predictions between these two models for a given sample indicates that the sample is atypical. Unlike previous methods, SAMIS-P doesn’t require knowing the true label before selection, making it practical for active learning.
SAMOSA operates in a two-step process. First, it uses a “distinguisher” neural network to filter out samples predicted as unknown from the unlabeled pool. Then, from the remaining known-class samples, it selects the most atypical ones (those with the highest SAMIS-P scores) for labeling. Interestingly, even some “invalid” (unknown class) samples selected by SAMOSA can be beneficial, as they help the distinguisher network improve its ability to filter out unwanted classes in subsequent rounds.
Extensive experiments demonstrate SAMOSA’s superior performance. Across several datasets, including CIFAR10, CIFAR100, and TinyImageNet, SAMOSA achieved up to a 3% accuracy improvement over state-of-the-art methods without introducing significant computational overhead. The paper highlights that SAMOSA’s accuracy gains are particularly pronounced in more challenging scenarios with higher “mismatch ratios” (a larger proportion of unknown classes).
The researchers also found that SAMOSA is robust to label noise during annotation, a common real-world issue. This resilience is attributed to SAM’s inherent robustness to label noise. Furthermore, the study delves into the “precision vs. effectiveness” trade-off in active learning. While other methods might aim for high precision (selecting only known-class samples), SAMOSA prioritizes effectiveness by focusing on sample quality and informativeness, even if it means initially lower precision. The authors argue that sample quality is more crucial for model performance than the sheer quantity of labeled samples.
To further explore this, a variant called SAMOSA-L was introduced, which selects samples with the lowest SAMIS-P scores, effectively prioritizing typical samples to build a larger dataset of valid samples. While SAMOSA-L excels at finding more known-class samples, SAMOSA (prioritizing atypical samples) consistently delivers higher model accuracy, especially in complex tasks.
Also Read:
- Enhancing AI Safety: A Training-Free Approach to Detecting Out-of-Distribution Data
- Optimizing LLM Fine-Tuning with Utility-Diversity Sampling
This research marks a significant advancement in open-set active learning, providing a theoretically grounded and empirically proven method for efficiently acquiring high-quality labeled data. The source code for their experiments is available for further exploration. For more details, you can refer to the full research paper: SAMOSA: Sharpness-Aware Minimization for Open-Set Active Learning.
