TLDR: TASK PGM is a novel framework that systematically optimizes the composition of fine-tuning data for large language models (LLMs). It uses an energy-based probabilistic model and behavioral divergences (like JSD and PMI) to quantify task relationships, balancing representativeness and diversity. This approach yields a closed-form solution, offering efficiency and interpretability. Empirical results show consistent performance improvements on Llama-2 and Mistral across various benchmarks compared to traditional heuristic methods.
The performance of large language models (LLMs) after fine-tuning heavily depends on the specific mix of data used during this process. Traditionally, selecting the optimal combination of task datasets has been a manual and often trial-and-error endeavor, with practitioners frequently relying on simple strategies like uniform or size-based sampling. This ad-hoc approach can lead to suboptimal performance, inefficient resource use, and models that struggle to generalize or overfit to certain data types.
A new framework, called TASK PGM, aims to revolutionize this process by offering a principled and scalable solution for optimizing the data mixture. Instead of guesswork, TASK PGM systematically selects continuous task proportions by minimizing an ‘energy function’ over a Markov Random Field (MRF). Imagine tasks as nodes in a network, and the connections between them represent how they relate to each other.
What makes TASK PGM unique is how it models these task relationships. It doesn’t just look at what the tasks are about semantically. Instead, it uses ‘behavioral divergences’ – like Jensen-Shannon Divergence and Pointwise Mutual Information – which are calculated from how single-task fine-tuned models predict outcomes. This means it understands how tasks functionally interact, offering a much deeper insight than just comparing their topics.
The framework provides a clear, mathematical solution that inherently balances two crucial aspects: representativeness and diversity. Representativeness means favoring tasks that show broad usefulness and positive influence across the entire set of tasks. Diversity means avoiding redundancy by penalizing tasks that offer very similar capabilities. This balance ensures the fine-tuned model learns broadly without being bogged down by repetitive information.
TASK PGM offers several distinct advantages. It directly optimizes the continuous proportions of tasks, unlike other methods that might just select subsets. By using predictive distribution divergences, it captures how tasks truly interact at a functional level. The framework is also theoretically sound, offering a closed-form solution that avoids costly, iterative searches, making it efficient. Furthermore, the derived mixture weights and task affinities provide valuable insights into why certain data compositions work best, enhancing interpretability.
In experiments, TASK PGM has shown consistent improvements. When applied to Llama-2 and Mistral models, mixtures derived using TASK PGM consistently outperformed traditional uniform, size-proportional, and other advanced selection methods on benchmarks like MMLU and BIG-Bench-Hard. For instance, on MMLU, it achieved significant absolute improvements. Both Pointwise Mutual Information (PMI) and Jensen-Shannon Divergence (JSD) performed similarly well as similarity metrics, indicating the method’s flexibility.
The research also found that increasing the number of instances in the mixtures generally boosted performance on more complex benchmarks, especially with Mistral models. Heuristic methods, in contrast, often failed to generalize effectively as data complexity increased, highlighting the robustness of TASK PGM’s principled approach. The study also explored the impact of hyperparameters, finding that a balanced ratio between the unary (representativeness) and pairwise (diversity) terms was key to optimal performance.
Also Read:
- Bridging the Gap: How Supervised Fine-Tuning Connects to Reinforcement Learning and Can Be Enhanced
- Aligning Language Models: The Power of Inferring Rewards with Inverse Reinforcement Learning
This work provides a systematic, theoretically-grounded alternative to the often empirical and artistic process of mixing datasets for LLM fine-tuning. It promises improved performance, greater efficiency, and a deeper understanding of how to effectively train large language models. For more details, you can refer to the full research paper: Learning What Matters: Probabilistic Task Selection via Mutual Information for Model Finetuning.
