TLDR: The RECAP research introduces a new benchmark and method for ‘intent rewriting’ in conversational AI. It addresses the challenge of ambiguous user intent in LLM-powered multi-agent systems by rephrasing dialogues into clear, concise user goals. The paper demonstrates that prompt-based and DPO-trained rewriters significantly improve the quality of AI agent plans, especially in complex conversational scenarios like intent shifts and multi-goal interactions. It also shows that LLM planners are highly sensitive to input quality and that fine-tuning rewriters with human preferences leads to superior performance.
In the evolving landscape of conversational AI, understanding what a user truly wants is paramount, especially when large language models (LLMs) are orchestrating multiple AI agents to complete complex tasks. However, real-world conversations are rarely straightforward; they can be ambiguous, incomplete, or change direction mid-way. This presents a significant challenge for AI systems trying to plan and execute tasks effectively.
A new research paper introduces RECAP (REwriting Conversations for Agent Planning), a novel benchmark and approach designed to tackle this very problem: intent rewriting. The core idea behind intent rewriting is to transform a messy, multi-turn user-agent dialogue into a clear, concise representation of the user’s most current and relevant goal. This clarified intent then serves as a much better input for the AI’s planning module, leading to more accurate and efficient task completion.
The Challenge of Understanding User Intent
Traditional methods for understanding user intent, often relying on predefined categories, struggle in the open-ended nature of modern LLM-powered assistants. Users might change their minds, provide vague instructions, or introduce irrelevant information. These natural conversational quirks can easily confuse an AI planner, leading to errors, a poor user experience, and ultimately, task failure. The RECAP paper highlights several common challenges:
- Underspecified Intent: When the user’s goal lacks sufficient detail.
- Noisy Input: Irrelevant or off-topic dialogue turns obscuring the main objective.
- Shifted Intent: The user changes their goal mid-conversation.
- Multi-Intent: Multiple distinct goals presented simultaneously or sequentially.
Introducing the RECAP Benchmark
To address the lack of specific benchmarks for intent rewriting in agent planning, RECAP was developed. This benchmark systematically captures a wide array of intent rewriting challenges across various domains like cooking, programming, health, flights, and restaurants. It includes conversations of different lengths (short, medium, long) and is designed to reflect realistic user-agent interactions. The dataset was synthetically generated using LLMs and carefully vetted by humans to ensure quality and coherence.
How Intent Rewriting Improves Planning
The researchers demonstrated that feeding raw conversation history directly to an LLM-based planner can lead to problematic plans. For instance, a planner might incorrectly interpret past suggestions as current requests or attempt to book a hotel when the user only asked for information. By contrast, an explicit, rewritten intent helps the planner focus on the user’s up-to-date and complete goals, resulting in higher-quality plans.
Evaluating the Rewriters
The study evaluated different types of rewriters:
- Dummy Rewriter: Simply uses the original conversation as is.
- LLM-based Basic Rewriter: Performs a generic summarization.
- Advanced Rewriter: A prompt-based LLM approach specifically instructed to create concise, unambiguous, and task-aware representations of the latest user intent.
- DPO-based Rewriters (DPO:human and DPO:LLM): These models were fine-tuned using Direct Preference Optimization (DPO). DPO:human leveraged human judgments on plan quality, while DPO:LLM used preferences generated by an LLM-as-a-judge evaluator.
The evaluation used a combination of structural metrics (like differences in plan nodes and edges), semantic metrics (BERTScore), and crucially, human preference judgments. Human annotators, and later fine-tuned LLM evaluators, assessed plans based on criteria such as reflecting the latest intent, avoiding fabrication, task granularity, completeness, and logical order.
Also Read:
- Structuring Intelligence: Language Models Crafting Hierarchical Learning Environments for AI Agents
- Enhancing Conversational Search Through Iterative Clarification and Rewriting
Key Findings
The research revealed several important insights:
- LLM planners are highly sensitive to the quality of the input intent, with differences in plan structure becoming more pronounced in longer, more complex conversations.
- The Advanced rewriter consistently produced plans that were preferred by human annotators, especially in challenging scenarios involving shifted or multi-intents.
- Fine-tuning rewriters with human preference labels (DPO:human) yielded the best results, significantly improving planning utility and outperforming the prompt-based Advanced rewriter. This highlights the value of human-aligned supervision.
- LLMs can be effectively fine-tuned to act as reliable and cost-efficient evaluators for plan preferences, offering a scalable alternative to human annotation.
This work underscores intent rewriting as a crucial and practical component for enhancing agent planning in open-domain dialogue systems. By providing a clearer, more focused understanding of user goals, AI assistants can become more effective and adaptive. For more details, you can read the full research paper here.
