TLDR: This research introduces a multi-stage framework for generative query suggestion in conversational AI systems. It leverages prompt engineering for initial setup, supervised fine-tuning with user click data, and a novel Gaussian Reward Model (GaRM) to capture the uncertainty in user preferences. The final stage uses reinforcement learning with a composite reward function to align the AI’s generation policy with these preferences. The framework significantly improves user engagement and suggestion quality, achieving a 34% increase in click-through rate in live tests.
The landscape of human-computer interaction has been significantly transformed by the rise of conversational systems, such as chatbots and virtual assistants. A crucial feature in these systems is query suggestion (QS), which proactively recommends relevant questions or phrases based on the ongoing conversation. Traditionally, QS methods relied on selecting suggestions from a fixed pool of past queries. However, the emergence of Large Language Models (LLMs) has shifted this paradigm, enabling the generation of novel, contextually specific suggestions.
Despite the power of LLMs, aligning their generated outputs with the nuanced and often implicit preferences of users remains a significant challenge. To address this, researchers from Bytedance – Junhao Yin, Haolin Wang, Peng Bao, Ju Xu, and Yongliang Wang – have introduced a novel multi-stage framework designed for progressively aligning the AI’s generation policy with user intent. This framework systematically moves from initial broad signals to a more refined understanding of user preferences.
Also Read:
- Enhancing Large Language Model Reasoning Through Contrastive Learning and Reinforced Fine-Tuning
- Unlocking LLM Potential: A Seed-Free Approach to Instruction Tuning
The Multi-stage Alignment Framework
The proposed framework consists of four key phases:
1. Prompt Engineering for Cold-start: This initial phase focuses on carefully crafting prompts to guide the LLM in generating relevant and diverse suggestions. It serves as a starting point for the system and, crucially, helps collect the first batch of real-world user click data.
2. Supervised Fine-Tuning (SFT): Using the collected click data, the base LLM undergoes a two-stage fine-tuning process. First, a “single-suggestion model” is trained to generate high-quality individual queries. This model then helps curate a more complex dataset, which is used to fine-tune the main SFT model to generate a list of three suggestions. This distillation method on click logs creates a robust foundational model.
3. Preference Modeling: Recognizing that user choices are often non-deterministic and noisy, the researchers developed a Gaussian Reward Model (GaRM). Unlike traditional reward models that provide a single, deterministic score, GaRM represents user preferences as probability distributions. This approach effectively captures the inherent uncertainty in user intent, making the model more robust to the noise in preference data. The paper highlights an analytically tractable loss function with variance regularization for GaRM, which improves efficiency and stability.
4. Reinforcement Learning (RL): In the final stage, reinforcement learning is employed to optimize the generative policy directly against the refined preference understanding from GaRM. To ensure stable policy updates and mitigate the risk of “reward hacking” (where the model exploits flaws in the reward signal), a composite reward function is designed. This function integrates GaRM’s probabilistic scores with auxiliary signals, including rule-based heuristics (like format, length, diversity, and language consistency checks) and prompt-based quality metrics. A novel out-of-distribution regularization method and a two-stage reward fusion technique further enhance training stability.
Extensive experiments, including live A/B tests on a production conversational AI system named Cici, demonstrated the framework’s effectiveness. It significantly outperformed baseline models on both automatic and human evaluations. Notably, the framework achieved a 34% relative increase in user engagement, as measured by click-through rate.
The research underscores the importance of a systematic, multi-stage approach to align generative AI models with complex human preferences. By moving from coarse behavioral signals to a nuanced and robust model of user intent, this framework establishes a virtuous cycle where improved models generate higher-quality data for future training iterations, ultimately leading to more intuitive and satisfying conversational AI experiences.
For a deeper dive into the technical details, you can access the full research paper here: From Clicks to Preference: A Multi-stage Alignment Framework for Generative Query Suggestion in Conversational System.
