TLDR: COSMOS is an LLM-powered simulator that evaluates online content moderation strategies by running parallel “factual” and “counterfactual” social network conversations. It reveals that personalized moderation is more effective, toxicity spreads through social contagion, and moderation impacts users differently based on psychological traits, primarily targeting extreme toxicity.
Online social networks (OSNs) are a vital part of modern communication, but they also grapple with the persistent challenge of toxic and abusive discourse. Content moderation is the primary tool platforms use to combat this, yet its true effectiveness has remained difficult to measure. The high cost of data collection, API restrictions, and the inherent difficulty in controlling experimental variables in real-world settings make rigorous evaluation a significant hurdle.
A new research paper introduces a novel approach to this problem: leveraging Large Language Models (LLMs) to power counterfactual simulations of OSN conversations. This method, implemented in a simulator called COSMOS (COunterfactual Simulations of MOderation Strategies), aims to provide a controlled environment for evaluating moderation strategies by generating empirical evidence rather than solely collecting it from the real world.
What is COSMOS?
COSMOS is an LLM-powered simulator designed to mimic online social network conversations. It features LLM-based agents, each with distinct socio-demographic and psychological profiles, capable of interacting within an OSN-like environment. The core innovation of COSMOS lies in its ability to run two parallel simulations simultaneously: a “factual” simulation where agents interact freely, and a “counterfactual” simulation where moderation interventions are applied, with all other variables kept constant. This parallel setup allows researchers to directly observe and measure how specific moderation strategies influence toxic behavior.
The simulator focuses on conversational dynamics, meaning it simulates posts and comments but excludes actions like likes, follows, or re-posts to maintain focus and reduce variability. Agents in COSMOS are equipped with profile modules (containing demographic and psychological traits), sensory modules (for environmental input), and memory modules (to store past moderation messages). They can choose to post, comment, or do nothing, with their actions and content generation influenced by their profiles, sensory input, and memory of past moderation.
How Moderation is Simulated
COSMOS integrates various moderation strategies. When an agent generates toxic content in the counterfactual simulation (detected by a state-of-the-art toxicity detector like Google’s Perspective API), moderation is activated. The paper explores two main types of interventions:
- Ex Post (BAN): This strategy involves banning agents if their content violations exceed a certain tolerance level. Banned agents are then unable to generate further content in the counterfactual feed.
- Ex Ante (Preventative): These interventions aim to prevent recidivism. They include:
- One-Size-Fits-All (OSFA): A generic, default moderation message is delivered to the agent.
- Personalized Moderation Intervention (PMI): Moderation messages are tailored to the agent’s specific socio-psychological profile and their toxic post or comment. The paper explores different tones for PMI: Neutral, Empathizing, and Prescriptive.
A crucial aspect of COSMOS is its ability to model both direct effects (on the moderated agent’s content) and indirect, cascading effects of moderation, where changes in one agent’s behavior influence subsequent interactions from other agents in the conversation thread.
Key Findings from COSMOS Simulations
The extensive experiments conducted with COSMOS yielded several significant insights:
- Psychological Realism: The simulated toxic behavior was found to be believable and consistent, showing significant correlations between toxicity and psychological traits (e.g., low agreeableness, low conscientiousness, high neuroticism) that mirror real-world observations.
- Toxicity Contagion: The simulations demonstrated that toxic behavior propagates across conversation threads, with a significant correlation between the toxicity of parent and child nodes. This highlights the social contagion aspect of online toxicity.
- Personalized Moderation is More Effective: Among the ex ante strategies, Personalized Moderation Interventions, particularly the Neutral variant (PMI-N), showed superior effectiveness in reducing toxicity. This suggests that adaptable, tailored moderation messages are more impactful than generic ones.
- Deplatforming Effects of Low Tolerance Bans: While banning agents (BAN) significantly reduced overall toxicity, especially with low tolerance levels, it came at the cost of substantial content loss, including a notable fraction of non-toxic content. This highlights the trade-off between mitigation and deplatforming.
- Sensitivity to Psychological Traits: Moderation strategies had comparable effects on similar personality types, successfully targeting agents typically associated with toxic behavior (low agreeableness, high neuroticism, low conscientiousness).
- Impact on Extreme Toxicity: Moderation primarily affected extreme toxic behavior (the highest quantiles of toxicity), with varying effects on milder forms of toxicity.
Also Read:
- AI Agents Reveal How Mental Schemas Shape Misinformation Responses
- New AI System ED2D Uses Evidence-Based Debate to Combat Misinformation
Limitations and Future Directions
The researchers acknowledge several limitations. LLMs can sometimes “hallucinate” or produce unexpected, unbelievable content, which required specific handling in the simulations. The current version of COSMOS focuses solely on conversations, excluding other OSN dynamics like follows or reactions, though future work plans to incorporate these. The computational cost of LLM-based simulations also poses a challenge for scaling to real-sized populations. Finally, the inherent biases of LLMs, derived from their training data, are a concern that needs further investigation.
COSMOS represents a significant step forward in evaluating online content moderation. By providing a controlled, LLM-powered simulation environment, it offers a valuable complement to traditional field observations, enabling deeper insights into the complex dynamics of online toxicity and the effectiveness of various moderation strategies. For more details, you can read the full research paper here.
