TLDR: A new method called “Weighted Task Diversity” improves the efficiency of finetuning Large Language Models (LLMs) by strategically selecting data for annotation. Instead of focusing on prompt diversity, it prioritizes tasks where the LLM is less confident, while ensuring all tasks are represented. This approach significantly reduces the amount of human annotation needed (up to 80% cost reduction) while achieving comparable or even better model performance than traditional methods.
Large Language Models (LLMs) have shown incredible abilities across many different areas. However, making these powerful models work well for specific, specialized jobs often needs a lot of human help in the form of annotations. This process is usually slow, requires a lot of effort, and can be very expensive, creating a big hurdle for developing high-performing models for niche applications.
A new research paper, titled Improving Task Diversity in Label Efficient Supervised Finetuning of LLMs, tackles this challenge by focusing on a smarter way to select data for supervised finetuning (SFT). The authors, Abhinav Arabelly, Jagrut Nemade, Robert D. Nowak, and Jifan Zhang from the University of Wisconsin–Madison, propose a method that uses ‘task-diversity’ as a core principle for efficient data selection. This is a fresh approach, differing from existing methods that often focus on the diversity of individual prompts.
The core of their approach is built on two simple but powerful observations: First, task labels for different prompts are often already available. For example, a dataset might categorize prompts as ‘medical knowledge’ or ‘mathematics’. Second, pre-trained LLMs have varying levels of confidence when dealing with different tasks. Some tasks might be inherently harder for the model, leading to lower confidence in its predictions.
Combining these insights, the researchers developed a straightforward yet highly effective sampling strategy. They select examples across different tasks by giving more weight to tasks where the model shows less confidence. This ‘inverse confidence weighting’ means that tasks the model struggles with will receive more attention and, consequently, more annotated examples. The beauty of this method is its simplicity, making it much easier to implement and less computationally demanding compared to more complex sampling procedures.
The experimental results are quite impressive. Models trained using this new method performed as well as, or even better than, those trained with more complicated sampling techniques. In a notable finding, their method even achieved better accuracy than training on the complete dataset, showing a 4% increase in the MMLU (Massive Multitask Language Understanding) score. Across various annotation budgets and two different instruction finetuning datasets (FLAN V2 and Dolly), their algorithm consistently matched or surpassed the best existing methods, all while potentially reducing annotation costs by up to 80%.
Understanding the Approach
The paper addresses the ‘label-efficient learning problem’ for SFT, where the goal is to reduce the amount of human annotation needed while still improving model performance. Unlike traditional data selection, which assumes ground-truth responses are already known, label-efficient learning operates without these initial annotations.
The researchers define tasks based on their original data curation sources, such as different domains (e.g., medical, math) or desired skills (e.g., summarization, translation). For instance, the FLAN dataset includes data from over a thousand sources across 1,691 tasks, and the Dolly dataset covers eight distinct tasks like brainstorming and information extraction.
They introduce two main algorithms:
Task Diversity
This baseline method aims to allocate a roughly equal number of examples to each task, ensuring broad coverage across all defined tasks. It uses a round-robin sampling algorithm to distribute the annotation budget evenly.
Weighted Task Diversity
This is the more advanced method. It starts by allocating a small base budget to each task to ensure minimum coverage. The remaining budget is then distributed based on the inverse of the average model confidence for each task. Tasks where the pre-trained model is less confident (meaning it’s more uncertain about its answers) receive a larger share of the annotation budget. This ensures that the model learns more from areas where it needs the most improvement.
Experimental Validation
The experiments were conducted using the LLaMA-2 7B language model and evaluated on standard benchmarks like MMLU, BBH (Big-Bench Hard), and AlpacaEval. On the MMLU benchmark, the Weighted Task Diversity method significantly outperformed traditional sampling methods. For example, with a 45K budget, models trained with this strategy achieved performance similar to models trained on the full 90K dataset, effectively halving the annotation cost.
On the Dolly dataset, their method achieved the highest MMLU scores at both 3K and 6K budgets, even outperforming training on all examples by over 4% and saving 80% in annotation budget. Furthermore, evaluations using GPT-4 as a judge (AlpacaEval) showed that models trained with Weighted Task Diversity consistently achieved higher ‘win rates’ compared to baselines, even when using significantly fewer examples.
A qualitative analysis revealed that tasks like ‘open QA’ (open-ended question answering) received a majority of the allocated samples. This makes sense because open-ended questions are typically more challenging for LLMs, leading to lower model confidence and thus a higher allocation of annotation resources. Simpler tasks like classification still received some annotations, ensuring overall data coverage.
Also Read:
- Bridging the Creativity Gap in Large Language Models
- Optimizing Large Language Models: A Look at Efficient Attention Strategies
Conclusion
This research highlights the significant value of combining task structure with model uncertainty for efficient and scalable instruction tuning of large language models. By strategically selecting data based on task diversity and model confidence, the proposed methods offer a practical way to optimize the annotation process without sacrificing model performance, making the development of specialized LLMs more accessible and cost-effective.
