TLDR: A new research paper introduces two novel datasets (ML-DDD and AO3) that combine user-content interaction data with user-generated content warnings. These datasets enable quantitative evaluation of how recommendation systems handle sensitive or harmful content, moving beyond traditional engagement metrics. Preliminary analysis reveals that personalized recommendation algorithms tend to de-amplify sensitive content compared to non-personalized baselines, highlighting a promising direction for developing safer AI systems that prioritize user agency.
Personalized artificial intelligence (AI) systems, such as recommendation engines and chatbots, have become ubiquitous in how we consume content. These systems learn our preferences and deliver tailored experiences. However, a growing concern revolves around their potential to expose users to sensitive or harmful material, which can negatively impact overall well-being.
Traditionally, these recommendation systems are designed to maximize user engagement. While this objective can be beneficial, it can also lead to unintended adverse outcomes. For instance, there have been documented cases of fertility-related advertisements being shown to infertile women who tried to opt out, and eating disorder content being algorithmically recommended on platforms like TikTok. While academic research has extensively explored issues of bias and fairness in AI, the specific challenge of sensitive or harmful content in recommendations has received less quantitative attention.
To address this critical gap, a recent research paper titled “Datasets for Navigating Sensitive Topics in Recommendation Systems” by Amelia Kovacs, Jerry Chee, Kimia Kazemian, and Sarah Dean introduces two innovative datasets. These datasets are designed to provide relevant sensitivity labels for content, allowing researchers to evaluate personalized systems beyond just how much users engage with them. The goal is not to ban sensitive content outright, but to empower users with the agency to avoid content they find objectionable, such as animal deaths in movies, if they choose.
Introducing Novel Datasets
The researchers propose two novel datasets that augment standard user-content preferences with explicit content warning labels:
-
MovieLens and Does the Dog Die? (ML-DDD): This dataset integrates MovieLens rating data, a well-known collection of user movie ratings, with content warnings sourced from doesthedogdie.com. This community-driven website allows users to vote on whether specific warnings (e.g., “Is there blood/gore,” “Does an animal die”) apply to movies, TV shows, and other media. The ML-DDD dataset contains over 32,000 movies, 162,000 users, and nearly 23 million interactions, with 137 distinct content warnings.
-
Archive of Our Own (AO3): This dataset combines fan-fiction interaction data from Archive of Our Own (AO3), a popular fan-fiction repository, with user-generated warnings from the Webis Trigger Warning Corpus. AO3’s unique tagging system, maintained by its users, often includes labels that serve as warnings for potentially triggering content. This dataset is larger, featuring over 300,000 works, 1.3 million users, and 45 million interactions, with 36 distinct trigger warning categories. Unlike ML-DDD’s explicit ratings, AO3’s interactions are implicit, based on actions like giving “kudos” (likes).
These datasets are crucial because they provide a systematic way to measure a form of user harm related to unwanted exposure to sensitive content. By leveraging user-generated content warnings, the research focuses on ensuring user agency rather than imposing censorship.
Analyzing Sensitivity and Popularity
The paper conducts descriptive analyses on these new datasets, investigating the relationship between sensitive content and popularity. For ML-DDD, popularity is measured by average movie ratings, while for AO3, it’s measured by the number of “kudos” received. The findings indicate a clear shift in popularity distributions for works marked with content warnings. Permutation tests revealed that for 70% of the top warnings in both datasets, there is a statistically significant correlation between the presence of a warning and work popularity. Interestingly, 70% of ML-DDD labels and 50% of AO3 labels showed a negative correlation, meaning works with those warnings were systematically less popular.
The study also looked at how individual users interact with content based on warnings. For example, in ML-DDD, 57.30% of users rated movies without the “Is there blood/gore” label higher. In AO3, 72.60% of users interacted more with content labeled “pornography,” aligning with global trends observed in the dataset.
Understanding Warning Amplification
A key contribution of this research is the introduction of a novel metric called “Warning Amplification.” This metric measures, for a given user, the fraction of items in their top recommendations that have a specific warning, divided by the fraction of items in their history with that warning. A score of 0% amplification means the user receives the same amount of warnings in recommendations as in their history.
The researchers performed a preliminary analysis using both non-personalized (Random, TopPop) and personalized (Singular Value Decomposition for ML-DDD, Alternating Least Squares for AO3) recommendation algorithms. The results showed that:
-
For ML-DDD, the TopPop algorithm (which recommends popular items) produced the most amplification, while the personalized SVD algorithm produced the least. This is partly because many warnings correlate with popularity, and TopPop prioritizes popularity.
-
For AO3, the Random algorithm produced the most amplification, and the personalized ALS algorithm produced the least. Similar to ML-DDD, the personalized recommender consistently de-amplified sensitive content.
These findings suggest that personalized algorithms, when trained solely on interaction data, tend to de-amplify warnings compared to simpler baselines. This presents an intriguing area for future research to understand the underlying mechanisms of this de-amplification.
Also Read:
- Unlocking Reliability: How Statistical Methods Bolster Generative AI
- Reinforcement Learning: The Core Driver for Advanced AI Research Systems
Conclusion and Future Directions
This paper provides crucial datasets and a foundational analysis for understanding the complex interplay between user preferences, personalization, and sensitive content in recommendation systems. The preliminary results indicate that personalized algorithms do not amplify warnings as much as non-personalized ones, a finding that warrants further investigation.
The researchers hope these datasets will spur the community to explore several important directions, including a deeper understanding of how recommendation algorithms amplify or de-amplify sensitive content, investigating disagreements in user-generated sensitivity labels, and designing new algorithms that seriously consider sensitive content by modeling negative preferences or offering richer user controls. Addressing individual-level harms from unwanted sensitive content recommendations is a vital challenge for the future of AI systems. You can find more details about this research in the full paper available here.
