TLDR: A new framework, RLAAR (Curriculum Reinforcement Learning with Verifiable Accuracy and Abstention Rewards), addresses the ‘Lost-in-Conversation’ (LiC) problem in LLMs. LiC causes performance degradation in multi-turn dialogues due to premature answering. RLAAR trains models using dynamic multi-turn interactions, a mixed-reward system (for accuracy and informed abstention), and a competence-gated curriculum. This approach significantly improves both problem-solving accuracy in multi-turn settings and the model’s ability to correctly abstain when information is insufficient, leading to more reliable and trustworthy conversational AI.
Large Language Models (LLMs) have become incredibly adept at following instructions in single-turn interactions, acting as powerful problem solvers. However, a significant challenge arises in multi-turn conversations, a phenomenon dubbed “Lost-in-Conversation” (LiC). This occurs when instructions are revealed progressively over several turns, leading to a noticeable drop in performance and reliability. Instead of waiting for complete information, LLMs often jump to premature conclusions, polluting the conversation context with flawed assumptions and making it harder to integrate subsequent user clarifications.
To tackle this critical issue, researchers have introduced a novel framework called Curriculum Reinforcement Learning with Verifiable Accuracy and Abstention Rewards (RLAAR). This approach aims to train LLMs not only to generate correct answers but also to intelligently assess whether a question is solvable given the current information, and to abstain when it isn’t. This balance between problem-solving and informed abstention is key to building more reliable and trustworthy LLMs in interactive settings.
How RLAAR Works: A Multi-faceted Approach
RLAAR combines several innovative techniques to address the LiC problem:
1. Dynamic Multi-turn Rollouts: Unlike many existing reinforcement learning (RL) methods that treat prior turns as a fixed context for a single step, RLAAR simulates entire conversations dynamically. In each training step, the model’s response at any turn becomes part of the state for the next, allowing it to explore and learn from the long-term consequences of its actions. This includes three types of rollouts:
- Solvable-Single: The model receives the full question at once and provides a single answer.
- Solvable-Multi: The question is revealed in shards over multiple turns, and the model must provide a final answer after all information is presented.
- Unsolvable-Multi: Similar to Solvable-Multi, but some crucial information is intentionally withheld, requiring the model to recognize the unsolvability and abstain.
2. Mixed Rewards for Accuracy and Abstention: A standard RL setup that only rewards accuracy can inadvertently encourage premature answering. RLAAR introduces a mixed-reward system with two distinct, verifiable signals:
- Accuracy Reward: This is a standard reward for correctly solving a problem, applied in solvable scenarios. For math tasks, it checks if the answer matches the ground truth; for code tasks, it verifies if the generated code is executable and passes tests.
- Abstention Reward: Crucially, this reward is given when the model correctly identifies an unsolvable question and explicitly abstains (e.g., by outputting a predefined string like “Abstain”). This teaches the model the valuable meta-skill of knowing when it doesn’t know, preventing it from guessing and polluting the context.
3. Competence-Gated Curriculum Learning: Training an RL policy on long, multi-turn dialogues from scratch can be inefficient due to sparse rewards. RLAAR employs a curriculum learning strategy that gradually increases conversational difficulty, defined by the number of instruction shards. The training progresses through three stages:
- Threshold Establishment: Initial training on simple, single-turn problems to set a performance baseline.
- Main Training: Multi-turn tasks are introduced, starting with fewer turns (e.g., two shards). The model only progresses to more complex, longer conversations (more shards) once it reaches a predefined performance threshold.
- Randomized Training: Once the maximum difficulty is reached, the model is trained on conversations with a randomly sampled number of turns, promoting robustness and generalization.
Impressive Results on LiC Benchmarks
Evaluated on LiC benchmarks, RLAAR demonstrated significant improvements. For instance, the framework substantially mitigated LiC performance decay, improving scores from 62.6% to 75.1% on models like Qwen3-8B. Even more notably, it drastically improved calibrated abstention rates, jumping from 33.5% to 73.4%. This indicates that models trained with RLAAR are far better at recognizing when they lack sufficient information and choosing to abstain rather than providing incorrect answers.
The ablation studies further highlighted the importance of both the abstention reward and the curriculum learning strategy. Without the abstention reward, models still struggled with premature answering, even with multi-turn rollouts. Similarly, an improperly tuned or absent curriculum made training unstable and less efficient.
Also Read:
- Bridging Language and Numbers: How New AI Training Boosts LLM Reasoning
- SherlockLLM: Enhancing Search Through Intelligent Questioning
Towards More Trustworthy LLMs
The RLAAR framework offers a practical recipe for building multi-turn reliable and trustworthy LLMs. By explicitly training models to balance problem-solving with informed abstention through dynamic multi-turn interactions and a structured curriculum, it addresses a fundamental weakness in current LLM behavior. This research paves the way for conversational AI that can engage more naturally and reliably in complex, progressively revealed tasks, ultimately enhancing user trust and interaction quality. You can read the full research paper here.
