TLDR: This paper introduces “Retentive Relevance,” a novel survey-based metric for recommendation systems that directly asks users about their intent to return for similar content. It significantly outperforms traditional engagement signals and other surveys in predicting next-day user retention, especially for new or less active users. Integrated into a social media platform, it led to improved user retention, engagement, and content quality in A/B tests, promoting a more user-centered and responsible AI approach.
Recommendation systems are everywhere, guiding our daily digital experiences, from what videos we watch to what products we buy. Traditionally, these systems have relied heavily on immediate engagement signals like clicks, likes, and watch time to personalize content. However, a new research paper titled Retentive Relevance: Capturing Long-Term User Value in Recommendation Systems by Saeideh Bakhshi, Phuong Mai Nguyen, Robert Schiller, Tiantian Xu, Pawan Kodandapani, Andrew Levine, Cayman Simpson, and Qifan Wang, highlights a critical limitation: these short-term signals are often noisy, sparse, and don’t truly capture whether a user will be satisfied in the long run or continue using the platform.
The core challenge is transforming these fleeting interactions into reliable predictions of sustained user interest and long-term satisfaction. The authors point out that users who engage with content don’t always want more of the same, and engagement signals can be biased towards popular items, missing deeper, latent interests. This disconnect is particularly problematic when the goal is long-term user retention.
Introducing Retentive Relevance
To address this, the researchers introduce “Retentive Relevance,” a novel content-level, survey-based feedback measure. Unlike existing survey methods that focus on immediate satisfaction or interest, Retentive Relevance directly assesses a user’s forward-looking intent to return to the platform for similar content. The key question posed to users immediately after a recommendation is: “How likely or unlikely are you to return to [platform] to view more posts like this?” This question is designed to capture behavioral intentions that drive users back to platforms, which is crucial for long-term effectiveness.
Validation and Key Findings
The paper details a comprehensive evaluation, including offline analyses, large-scale production deployments, and A/B experiments. The findings consistently show that Retentive Relevance significantly outperforms both traditional engagement signals and other survey measures in predicting next-day user retention. This is particularly true for “low-signal users” – those with limited historical engagement data where conventional metrics are sparse or unreliable.
Through psychometric methods, Retentive Relevance demonstrated strong convergent and discriminant validity, meaning it correlates well with related concepts while remaining distinct from others. Crucially, it showed low mutual information with traditional engagement signals, indicating that it provides unique and complementary information that cannot be inferred from observed behavior alone.
In predictive modeling, incorporating Retentive Relevance led to substantial improvements in accuracy and ROC AUC for next-day retention prediction. For the overall sample, accuracy increased by 5.0 percentage points, and for low-signal users, ROC AUC improved by 0.070 points. Feature importance analysis further revealed that user responses to Retentive Relevance were among the strongest predictors of retention, often surpassing traditional engagement factors like likes, shares, and comments.
Real-World Impact and Production Integration
The researchers didn’t stop at theoretical validation; they developed a production-ready proxy model to integrate Retentive Relevance into a multi-stage ranking system on a major social media platform. This involved translating survey insights into real-time predictions and using calibrated score adjustments to boost content with high predicted retention intent and demote low-quality content.
Large-scale online A/B experiments demonstrated the practical impact. The group exposed to the Retentive Relevance-integrated system showed consistent improvements across key platform metrics:
- User retention (sessions per user) increased.
- Engagement activity (communication activity, like rates) improved, and skip rates decreased.
- Content quality significantly improved, with reductions in reported content, negative feedback, and “not interested” signals.
These results indicate that optimizing for Retentive Relevance creates a natural alignment between an improved user experience, platform growth, and better content quality.
Also Read:
- Enhancing Survey Insights: A New Framework for Evaluating Human Responses
- Enhancing Recommendations with LLM Agents: Bridging Reasoning and Scalability
Implications for AI and User Experience
This work establishes a user-centered paradigm for recommender systems, moving beyond short-term engagement to focus on long-term user value and sustained retention. By directly incorporating user intent, algorithmic decisions become more interpretable and aligned with individual values, contributing significantly to responsible AI development. The framework is scalable and broadly applicable to other AI systems where user feedback and intent can optimize complex models, offering a path to resolve traditional trade-offs between growth and responsibility.
