TLDR: Instruction-tuned Large Language Models (LLMs) often produce less diverse outputs, a phenomenon termed the ‘diversity gap,’ primarily due to Direct Preference Optimization (DPO). Researchers introduce ‘conformative decoding,’ a new strategy that leverages an LLM’s more diverse base model to guide the instruction-tuned model, successfully reintroducing output diversity while maintaining or improving quality, particularly for creative tasks like narrative generation.
Large Language Models (LLMs) have become incredibly powerful tools, capable of understanding and generating human-like text for a vast array of applications. However, a recent study highlights a significant challenge: while instruction-tuning makes these models more useful for specific tasks, it often reduces the diversity of their outputs. This reduction, termed the “diversity gap,” has important implications, especially for creative tasks like narrative generation.
The research paper, titled Mind the Gap: Conformative Decoding to Improve Output Diversity of Instruction-Tuned Large Language Models, delves into this issue, investigating why instruction-tuned LLMs produce less varied content than their foundational, or “base,” models. The authors, Max Peeperkorn, Tom Kouwenhoven, Dan Brown, and Anna Jordanous, specifically examined this phenomenon in the context of a narrative generation task, using various open-weight and open-source LLMs.
Understanding the Diversity Gap
The study confirms that instruction-tuning indeed leads to a significant drop in output diversity across different LLMs. This was measured using various metrics, including lexical and semantic diversity scores. While instruction-tuned models often show improved quality (generating more realistic and accurate texts), they tend to produce a narrower range of responses, indicating a trade-off between quality and diversity.
A key finding of the paper is the identification of the primary culprit behind this diversity loss: Direct Preference Optimization (DPO). The researchers meticulously analyzed the impact of different fine-tuning stages on models like OLMo, revealing that DPO, a method used to align models with human preferences, has the most substantial negative effect on output diversity. Supervised Fine-tuning (SFT) also contributes to the loss, but to a lesser extent, while Reinforcement Learning with Verifiable Rewards (RLVR) had minimal impact.
Furthermore, the study observed that using conversation templates, which are common for instruction-tuned models, can further restrict the output space, leading to even lower diversity and sometimes quality compared to simpler completion prompts.
Introducing Conformative Decoding
Motivated by these findings, the researchers propose a novel decoding strategy called “conformative decoding.” The core idea is to guide an instruction-tuned model using its more diverse base model. This strategy aims to reintroduce the lost diversity while still leveraging the instruction-following capabilities of the fine-tuned model.
In practice, conformative decoding works by combining the probability distributions of the instruct model and its base model. This weighted sum pushes the instruct model to “conform” to the broader output possibilities offered by the base model. It’s crucial to use this strategy in conjunction with a truncation method (like nucleus sampling) to prevent the generation of low-quality or nonsensical outputs.
Promising Results
Experiments with conformative decoding showed significant improvements in output diversity for most models tested, including Gemma 2, Mistral, and OLMo. Importantly, these improvements were achieved while maintaining or even slightly enhancing the quality of the generated narratives. This suggests that it is possible to bridge the diversity gap without sacrificing the benefits of instruction-tuning.
While the improvements were modest, they are significant, especially considering that the method still relies on the instruct model’s truncated distribution. The study notes that the effectiveness of conformative decoding might vary depending on the initial diversity gap of the model, with models exhibiting lower diversity benefiting more.
Also Read:
- Navigating the Future of AI: A Comprehensive Look at Language Model Alignment and Safety
- AI Models Learn to Fix Their Own Flawed Instructions
Future Implications
The findings of this research are crucial for the development of more versatile and creative LLMs. The ability to reintroduce diversity is particularly valuable for applications in creative writing, content generation, and agent-based simulations, where varied outputs are highly desirable. This work opens new avenues for future research, including exploring different truncation strategies and adapting the method for larger LLMs and diverse domains.
