TLDR: A new research paper explores how human annotators evaluate the safety of AI-generated images, finding that their judgments go beyond simple predefined categories. Annotators use moral, emotional, and contextual reasoning, often perceiving harm to others more than to themselves. The study highlights that factors like image quality and prompt-image mismatches also influence perceived harm, suggesting that current AI safety evaluation frameworks need to be more flexible and incorporate subjective human insights.
Understanding what makes AI-generated content safe is a complex challenge, far beyond simply checking off boxes. While developers often rely on structured rules and categories, real-world safety judgments are deeply influenced by personal, social, and cultural perceptions of harm. A recent research paper, titled “Just a strange pic”: Evaluating ‘safety’ in GenAI Image safety annotation tasks from diverse annotators’ perspectives, delves into this complexity by examining how human annotators evaluate the safety of AI-generated images. [RESEARCH_PAPER_URL: https://arxiv.org/pdf/2507.16033]
The study, conducted by researchers from Google Research and Google DeepMind, including Ding Wang, Mark D´ıaz, and Lora Aroyo, analyzed 5,372 open-ended comments from 637 diverse annotators. These annotators were tasked with assessing the safety of 1,000 AI-generated images paired with prompts from the Adversarial Nibbler dataset. The goal was to understand the qualitative reasoning behind their judgments, especially when their insights extended beyond predefined safety categories like bias, sexual explicitness, or violence.
What Annotators Really Consider
The findings reveal that annotators consistently bring moral, emotional, and contextual reasoning to their evaluations, which often goes uncaptured by standard safety frameworks. For instance, they frequently reflected on potential harm to others more than to themselves, grounding their judgments in lived experience, collective risk, and sociocultural awareness. This suggests that safety isn’t just about individual perception but also about broader societal vulnerability.
A significant insight was how the task structure itself, including annotation guidelines, influenced how annotators interpreted and expressed harm. Guidelines not only affected which images were flagged but also the moral judgment behind their justifications. Annotators often cited factors like image quality, visual distortion, and mismatches between the prompt and the generated output as contributing to perceived harm, aspects frequently overlooked in standard evaluation frameworks.
The Emotional Landscape of Safety
The study found that annotators expressed a wide range of emotions when evaluating images, including fear, anger, sadness, disgust, amusement, confusion, and uncanniness. For example, distorted faces or images suggesting violence often evoked fear, while stereotypical or offensive content triggered anger. Images depicting poverty or potential harm to children elicited sadness. Even images not explicitly flagged as harmful could evoke strong emotional responses, highlighting a critical need for evaluation frameworks to integrate these subjective and emotional elements.
Beyond the Image: Prompt Intent and Quality
Annotators didn’t just look at the generated image; they also considered the safety implications of the original prompt. They often distinguished between a harmful prompt leading to a benign image, or vice versa, indicating a dual evaluation process that current task designs often miss. Furthermore, image quality was deeply intertwined with safety judgments. Distortions or visual glitches were often described as “disturbing” or “indicative of harm,” rather than just technical flaws. This shows that annotators interpret quality artifacts as semantically meaningful, directly impacting their perception of harm.
Safety for Whom?
The research also explored how annotators judged harm for themselves versus for others. While most annotators rated content as more harmful to others than to themselves, this difference varied across demographic groups. For example, women and non-White annotators showed different patterns in their harm scores, suggesting that social experiences inform broader reasoning differences and sensitivity to harm, even when the content doesn’t directly depict their identity.
Also Read:
- Exploring Varied Human Views on Text-to-Image Model Safety
- Bridging the Gap: How AI Models Can Learn from Their Own Mistakes in Image Generation
Recommendations for a More Nuanced Approach
Based on these findings, the paper proposes several recommendations for improving AI safety evaluation tasks:
- Balancing Structure and Subjectivity: Task designs should allow for subjective interpretations and open-ended feedback alongside structured responses.
- Integrating Safety and Item Quality: Explicitly evaluating safety and quality in tandem can help disentangle these dimensions and provide a clearer understanding of their interplay.
- Expanding Frameworks: Guidelines should include diverse cultural and emotional interpretations of harm, and explicitly distinguish between “self” and “other” harm.
In conclusion, the research highlights that existing AI safety pipelines miss critical forms of reasoning that human annotators bring to the task. It argues for evaluation designs that encourage moral reflection, differentiate types of harm, and make space for subjective, context-sensitive interpretations of AI-generated content, ultimately leading to safer and more ethically aligned generative AI systems.
