TLDR: UserBench is a new interactive environment designed to evaluate how well AI agents collaborate with users, especially when user goals are vague or evolve. It simulates realistic, multi-turn interactions where agents must proactively clarify implicit preferences and use tools. Evaluations show current AI models struggle significantly with user alignment and preference elicitation, highlighting a gap between task completion and true user understanding. The benchmark aims to foster development of more collaborative and user-centric AI.
Large Language Models (LLMs) have shown remarkable abilities in complex tasks like reasoning and using tools. However, a significant area that remains underexplored is their capacity to actively work with users, especially when user goals are unclear, change over time, or are expressed indirectly.
To address this crucial gap, researchers have introduced UserBench, a new benchmark designed to evaluate how well AI agents interact with users in multi-turn, preference-driven conversations. UserBench features simulated users who begin with vague goals and gradually reveal their preferences. This setup forces AI agents to proactively ask clarifying questions and make informed decisions using various tools.
The core idea behind UserBench is to move beyond simply evaluating an agent’s ability to complete a task. Instead, it focuses on whether the agent truly understands and aligns with the user’s evolving needs. Human communication is often ambiguous; users don’t always state their full intent upfront, their goals can change during a conversation, and they might express preferences subtly or indirectly. UserBench is built to simulate these real-world communication traits: underspecification (vague initial goals), incrementality (preferences emerging over time), and indirectness (subtle cues).
Built on the standard Gymnasium framework, UserBench provides a flexible and expandable environment. It includes a standardized way for agents to interact and a stable backend for tool use, ensuring evaluations are consistent and repeatable. The benchmark uses travel planning tasks as its primary domain, where users implicitly reveal their preferences for flights, hotels, apartments, car rentals, and restaurants.
The data for UserBench is gathered with realism, diversity, and implicitness in mind. It includes hundreds of distinct preferences, each paired with multiple natural, indirect ways of expressing that intent. For example, a preference for direct flights might be expressed as, “I always keep my schedule packed tight, so I prefer travel routes that minimize transit time.” These preferences are then randomly combined to create over 10,000 unique travel scenarios, categorized by difficulty.
UserBench also incorporates tool augmentation, simulating database searches for each travel aspect. These tools provide pre-generated options, including “correct” (satisfying preferences), “wrong” (violating preferences), and “noise” (irrelevant or unrealistic) options. This design ensures controlled outputs and focuses the evaluation on user-centric reasoning rather than real-time search challenges. The environment simulates an “oracle user” who knows all preferences but only reveals them implicitly, either when the agent asks a relevant question or after a set number of turns without progress.
Agents in UserBench can perform three types of actions: “search” (querying the database), “action” (communicating with the user to clarify intent), and “answer” (recommending options). The environment evaluates these actions, providing feedback and revealing preferences implicitly.
Initial evaluations of various leading AI models, both open-source and closed-source, using UserBench have revealed significant findings. Models often provide answers that fully align with all user intents only about 20% of the time. Even the most advanced models uncover fewer than 30% of all user preferences through active interaction. This highlights a considerable gap between an agent’s ability to complete tasks and its ability to truly understand and align with user needs.
The research indicates that current LLMs struggle significantly with interactively uncovering and acting on user preferences. For instance, performance drops by over 40% when models are restricted to selecting only one option per travel aspect, showing their difficulty in making optimal decisions. While models are generally good at using tools (over 80% valid search attempts), their ability to ask precise, preference-relevant questions (valid action attempts) is much lower. This suggests that understanding users is a harder challenge than simply executing tool use.
Interestingly, the study found that simply allowing more interaction turns does not consistently improve performance; in some cases, it even degrades. This implies that many models fail to effectively use extended conversations to elicit preferences or refine their understanding, often leading to repetitive or off-topic dialogue. The number of preferences per aspect was identified as a key driver of difficulty, with models struggling more when a single aspect involves many complex preferences.
Also Read:
- Unmasking LLM Agent Hallucinations: A New Benchmark for Interactive Environments
- Enhancing Conversational AI: A New Framework for Goal-Aligned User Simulators
UserBench is seen as a foundational step towards creating truly user-centric agents—AI systems that are not just efficient task executors but genuine collaborative partners. The environment is designed to be flexible for both evaluation and training, supporting various configurations and reward functions. This allows for detailed analysis of agent behavior and can help in training future models to balance efficiency (quick responses) with effectiveness (satisfying user needs), ultimately leading to more satisfying human-AI interactions. All code and data for UserBench are publicly available to support further research. You can find more details about UserBench and access the resources at the research paper.
