TLDR: A new research paper introduces MODEL CITIZENS, a dataset of social media posts with toxicity annotations from diverse identity groups, including conversational context. It reveals significant disagreements between “ingroup” and “outgroup” annotators, highlighting the subjectivity of toxicity. The paper also presents LLAMACITIZEN and GEMMACITIZEN, new models finetuned on this dataset, which significantly outperform existing state-of-the-art toxicity detectors by better reflecting community perspectives, advocating for community-informed content moderation.
In the ever-evolving landscape of online communication, ensuring safe and inclusive digital spaces is a paramount challenge. A new research paper, titled “ModelCitizens: Representing Community Voices in Online Safety,” delves into the complexities of automatic toxic language detection, a task that is inherently subjective and deeply influenced by community norms and individual experiences.
Traditional toxicity detection models often fall short because they simplify diverse annotator perspectives into a single ‘truth.’ This approach can overlook crucial context-specific nuances, such as the use of reclaimed language within certain communities, leading to unintended censorship or misidentification of harmful content.
To address this critical gap, researchers have introduced a groundbreaking dataset called MODEL CITIZENS. This dataset comprises 6,822 social media posts and an impressive 40,000 toxicity annotations, gathered from a wide array of identity groups. What makes MODEL CITIZENS unique is its focus on capturing the role of conversational context, a common feature of real-world social media interactions. The dataset augments posts with conversational scenarios generated by large language models (LLMs) to better reflect how toxicity unfolds in dialogue.
A key finding from the MODEL CITIZENS dataset is the significant disagreement between ‘ingroup’ and ‘outgroup’ annotators. Ingroup annotators are individuals who self-identify with the target group mentioned in a post, while outgroup annotators do not. The study found that ingroup and outgroup annotators disagreed on 27.5% of posts, with outgroup annotators more frequently labeling content as toxic—a phenomenon the researchers term ‘amplified harm.’ This highlights that perceptions of toxicity vary greatly depending on one’s lived experience and community affiliation.
The research also revealed that existing state-of-the-art toxicity detection tools, such as OpenAI Moderation API and GPT-o4-mini, underperform on the MODEL CITIZENS dataset. Their accuracy further degrades when faced with context-augmented posts, underscoring their misalignment with the perspectives of targeted communities.
To overcome these limitations, the researchers developed new models: LLAMACITIZEN-8B and GEMMACITIZEN-12B. These models are based on LLaMA and Gemma architectures and were finetuned using the MODEL CITIZENS dataset. The results are promising: LLAMACITIZEN-8B, for instance, outperformed GPT-o4-mini by 5.5% in in-distribution evaluations and showed improved accuracy across all identity groups. This demonstrates the profound importance of incorporating community voices directly into the design and training of AI systems for content moderation.
Also Read:
- Advancing Text Detoxification: A Data-Efficient Approach for Safer Online Discourse
- Standardizing LLM Evaluation: How Fine-Tuning Improves Model Rankings
This work lays a crucial groundwork for future research aimed at creating more inclusive and equitable toxicity detection models. By centering community perspectives, we can move towards online spaces that are truly safe and welcoming for everyone. For more details, you can read the full research paper here.
