TLDR: A systematic review of 23 studies on Conversational Recommender Systems (CRSs) reveals that user experience (UX) evaluation is fragmented, often focusing on satisfaction and usability while neglecting deeper emotional and relational aspects. Most evaluations rely on short, post-interaction surveys, failing to capture dynamic user experiences. While personalization is common, true adaptive behavior is less frequent and poorly evaluated. The emergence of LLM-powered CRSs introduces new challenges like transparency and control over verbosity, which current evaluation methods are ill-equipped to handle. The paper calls for more comprehensive, dynamic, and context-sensitive UX evaluation methods for future CRSs.
Conversational Recommender Systems, or CRSs, are interactive tools that help users find and select items through natural language conversations. Unlike older recommendation systems that just suggest items based on past behavior, CRSs allow for dynamic, multi-turn dialogues where users can express their preferences, explore options, and clarify their needs. The field has evolved significantly, especially with the rise of large language models (LLMs) like GPT-4, which enable more human-like and open-ended conversations.
Despite their growing presence in areas like e-commerce, media, travel, and health, the evaluation of user experience (UX) in CRSs has been quite limited. Many existing reviews focus on technical aspects such as natural language understanding and recommendation accuracy, often overlooking how users actually perceive and interact with these systems. This systematic review aimed to fill that gap by analyzing 23 empirical studies published between 2017 and 2025, following PRISMA guidelines.
Understanding User Experience in CRSs
The review explored several key questions, including what UX dimensions are evaluated, the methods and tools used for evaluation, how adaptivity and personalization influence the experience, and the unique challenges posed by LLM-powered CRSs.
The findings revealed that user satisfaction is the most frequently assessed UX metric, appearing in 18 out of 23 studies. Other common dimensions include personalization, usefulness, trust, ease of use, and recommendation quality. Interestingly, more emotional and reflective aspects like curiosity, self-reflection, and emotional support were less frequently reported, mainly appearing in studies involving LLM-powered systems.
The type of UX priorities also varied by domain. For instance, e-commerce CRSs often focused on transactional factors like purchase intention and privacy. Music and movie CRSs, on the other hand, emphasized enjoyment, novelty, and transparency. Nutrition systems looked at long-term goals such as loyalty and usage frequency.
Evaluation Methods and Their Limitations
Most UX evaluations in CRS research rely heavily on self-reported measures, typically post-interaction surveys with Likert scales. While these provide quick feedback, they often fail to capture the dynamic and evolving nature of user experience during a multi-turn conversation. Behavioral measures, such as tracking dialogue length or recommendation acceptance rates, were used in about half of the studies, but external measures (like third-party annotations) were rare.
Qualitative methods, such as interviews and diary studies, were used in only a few instances, limiting deeper insights into user perceptions. The review also noted that many studies either designed their own questionnaires or didn’t fully describe their instrument sources, which makes it harder to replicate findings and compare results across different studies.
Adaptivity and Personalization
The review classified systems based on whether they adapted their behavior dynamically during interaction. Seven studies were identified as fully adaptive, meaning they adjusted dialogue flow or content generation in real-time based on user input. Examples include systems that shifted initiative between proactive and reactive modes or adapted agent roles in group decision-making. Nineteen studies incorporated some form of personalization, but many used static methods, tailoring content based on pre-defined user traits rather than dynamically changing interaction logic.
Adaptive systems tended to assess more interaction-focused UX metrics like perceived control and engagement, often through experimental designs. Non-adaptive systems, however, typically relied on simpler post-hoc ratings of satisfaction or content relevance. This suggests that while personalization is common, true dynamic adaptivity that influences system behavior during a conversation is less frequent and often not thoroughly evaluated for its UX impact.
The Rise of LLM-Powered CRSs
Only two studies in the review explicitly implemented and evaluated LLM-powered CRSs. Both used OpenAI’s GPT architecture. While these systems offered increased fluency and expressivity, they also introduced new challenges. A major concern is the “epistemic opacity” of LLMs, meaning users often don’t understand how or why a recommendation was generated, especially without clear explanations or source attribution. This can erode trust, particularly if the system “hallucinates” or provides inaccurate information.
Another issue is the lack of user control over LLM responses, which can sometimes be overly verbose or inconsistent in tone. Current evaluations of LLM-based CRSs are limited, often relying on single-session interactions and lacking fine-grained, turn-level UX instrumentation to understand real-time user responses or breakdowns. The paper highlights that as LLM-based CRSs become more common, new evaluation methods are needed to address these unique challenges, including how users tolerate hallucinations or perceive conversational agency.
Also Read:
- Understanding Dialogue Systems Engineering: A Comprehensive Overview
- The AI Evolution in Virtual Worlds: How Language Models are Redefining VR Gaming
Looking Ahead for CRS User Experience
The review concludes that UX evaluation in CRSs is fragmented and often too narrow, focusing mainly on satisfaction and usability while neglecting deeper affective and relational dimensions. There’s a strong call for more sophisticated, context-sensitive, and longitudinal evaluation strategies that can capture the evolving nature of user experience in multi-turn dialogues. This includes developing validated, CRS-specific UX instruments, embedding in-situ and longitudinal designs, and incorporating diagnostic tools for analyzing LLM behaviors. The goal is to move towards systems that are not only accurate and efficient but also transparent, trustworthy, and truly engaging for users. For more detailed insights, you can read the full research paper here.
