TLDR: A new research paper introduces ImbLLM, an LLM-based oversampling method that significantly improves imbalanced classification by generating more diverse synthetic minority samples. It achieves this through three key innovations: a sampling strategy conditioned on both minority labels and features, a permutation strategy that fixes the target variable at the beginning of sentences during fine-tuning, and fine-tuning with minority and interpolated samples to avoid bias and enhance variability. Extensive experiments show ImbLLM outperforms state-of-the-art baselines in F1-score, AUC, and generates higher quality and more diverse synthetic data.
Imbalanced classification is a common challenge in machine learning, where some categories in a dataset have significantly fewer examples than others. This often leads to biased models that struggle to accurately predict the minority class. A widely used technique to combat this is oversampling, which involves generating additional samples for the underrepresented classes to balance the dataset.
Traditional oversampling methods, such as SMOTE, often require converting categorical data into numerical formats, which can lead to a loss of valuable information. While Large Language Models (LLMs) have recently emerged as a promising alternative, overcoming this limitation by treating tabular data as text, current LLM-based approaches typically generate synthetic minority samples with limited diversity. This lack of diversity can reduce the robustness and generalizability of machine learning models trained on these augmented datasets.
Introducing ImbLLM: Enhancing Diversity in LLM-based Oversampling
A new research paper, “Large Language Models for Imbalanced Classification: Diversity makes the difference,” introduces ImbLLM, a novel LLM-based oversampling method designed to significantly enhance the diversity of generated minority samples. Authored by Dang Nguyen, Sunil Gupta, Kien Do, Thin Nguyen, Taylor Braund, Alexis Whitton, and Svetha Venkatesh, ImbLLM addresses the diversity gap through three key innovations:
1. Diverse Sampling Strategy: Existing LLM methods for generating synthetic samples typically condition the generation process solely on the minority label (e.g., “Income is >=200K”). This can lead to very similar outputs, as the LLM might consistently pick the most probable next token. ImbLLM introduces a more diverse sampling strategy by conditioning synthetic sample generation on both the minority label and randomly selected features and their values (e.g., “Income is >=200K, Edu is Bachelor”). This creates a wider variety of initial prompts, leading to more diverse synthetic samples.
2. Improved Permutation Strategy: When fine-tuning LLMs for tabular data, existing methods often randomly shuffle both features and the target variable within a sentence. However, in decoder-only LLMs, an attention mask limits how tokens interact, meaning the minority label might not attend to all preceding features if it’s shuffled to the middle or end of the sequence. ImbLLM proposes a new permutation strategy that fixes the minority label at the beginning of the sentence while only permuting the features. This ensures the LLM can establish attention scores with all features, allowing for a broader range of predictions and thus more diverse synthetic samples.
3. Fine-tuning with Interpolated Samples: Current LLM-based methods often fine-tune the LLM using both majority and minority samples, which can still bias the model towards the majority class. Some methods attempt to verify generated samples using an external classifier, but this can be unreliable if the classifier itself is trained on an imbalanced dataset. ImbLLM fine-tunes the LLM exclusively on minority samples and, crucially, on interpolated samples. Interpolation helps to enrich the variability of continuous variables within the minority class, especially since the original minority dataset might be small. This approach ensures the LLM focuses on generating true minority samples without bias and eliminates the need for an unreliable verification step.
Empirical Validation and Theoretical Support
The researchers conducted extensive experiments on 10 real-world tabular datasets, comparing ImbLLM against eight state-of-the-art baselines, including traditional methods like SMOTE, generative models like CTGAN, and other LLM-based approaches. ImbLLM consistently demonstrated superior performance, achieving the best F1-score in five cases and ranking second in three others. On average, ImbLLM showed a significant improvement of approximately 3% over the runner-up method (CTGAN) and 8% over classifiers trained without oversampling.
Beyond classification performance, ImbLLM also excelled in generating high-quality and diverse synthetic samples. Metrics like “Close probability” indicated that ImbLLM’s synthetic samples were more similar to original minority samples, while “Coverage” scores confirmed their greater diversity, effectively capturing the variability of real minority data. Ablation studies further validated the effectiveness of each of ImbLLM’s proposed modifications.
The paper also provides a theoretical analysis from an entropy-based perspective, formally demonstrating that ImbLLM’s generation process promotes diversity. This theoretical grounding, combined with strong empirical results, highlights the method’s robustness and effectiveness.
Also Read:
- Enhancing Language Model Reasoning Through Representation-Based Exploration
- Bridging the Knowledge Gap: A New Approach to Fine-Tuning Language Models
Conclusion
ImbLLM represents a significant advancement in addressing imbalanced classification using large language models. By focusing on enhancing diversity through novel sampling, permutation, and fine-tuning strategies, it generates synthetic minority samples that are both realistic and highly varied. This leads to more robust and generalizable machine learning models, offering a powerful tool for tackling one of machine learning’s persistent challenges. You can read the full research paper here.
