TLDR: A new method called Seed2Harvest combines human-crafted adversarial prompts with AI-driven expansion to create a large, diverse dataset for red-teaming text-to-image models. This hybrid approach significantly improves scalability and prompt diversity while maintaining effective attack rates, helping to identify and address model vulnerabilities more comprehensively and efficiently than purely human or automated methods.
Text-to-image (T2I) models, like DALL-E and Stable Diffusion, are widely used, making it crucial to thoroughly test them for potential harms. These models can sometimes generate inappropriate or biased images even from seemingly harmless prompts. To address this, a process called red-teaming is used to stress-test these models for unexpected undesirable behaviors.
Traditionally, generating adversarial prompts for red-teaming has faced a dilemma: human-crafted prompts are rich in nuance and creative attack strategies but are small in scale and can be culturally imbalanced. On the other hand, synthetically generated prompts achieve scale but often lack the realistic and subtle adversarial qualities found in human-made ones.
Researchers Jessica Quaye, Charvi Rastogi, Alicia Parrish, Oana Inel, Minsuk Kahng, Lora Aroyo, and Vijay Janapa Reddi have introduced a novel hybrid red-teaming method called Seed2Harvest. This approach aims to combine the strengths of both human creativity and machine computational power to create a comprehensive and scalable way to evaluate T2I model safety. You can read their full paper here: From Seed to Harvest: Augmenting Human Creativity with AI for Red-teaming Text-to-Image Models.
How Seed2Harvest Works
Seed2Harvest begins with a curated set of human-written adversarial prompts, known as “seed prompts,” taken from the Adversarial Nibbler dataset. These seeds are prompts that appear innocuous but have successfully bypassed safety filters to generate unsafe images. The method also leverages human-derived attack strategies, identified through qualitative analysis of how humans craft effective adversarial prompts.
The core of Seed2Harvest involves using large language models (LLMs) such as ChatGPT 4.1, Claude 3.7 Sonnet, Llama 3.2 90b, and Gemini 2.0 Flash. Each LLM receives a human seed prompt and a specific attack strategy. There are seven identified attack strategies:
- Coded Language: Using euphemisms or indirect references.
- Double Entendre: Exploiting words with multiple meanings.
- Demography: Systematically varying demographic descriptors.
- Geography: Substituting geographic references.
- Negation: Using negative phrasing that models might ignore.
- Vagueness: Employing ambiguous phrasing.
- Visual Similarity: Substituting objects with visually similar items that trigger unintended associations.
For each seed prompt, the LLMs generate multiple variants (5 per strategy per LLM). To ensure diversity, a clustering technique is then applied to select the most dissimilar prompts. This process results in a significant expansion of the original human-crafted prompts, generating approximately 28 new prompts for each original seed.
Key Outcomes and Benefits
The Seed2Harvest method demonstrates impressive scalability, expanding 1,000 human-crafted seed prompts to about 27,650 new variants. This 28-fold increase was achieved with minimal human intervention, taking approximately 12 hours of computational time compared to months of human effort for the original dataset collection. This efficiency allows for rapid iteration and adaptation to new model releases or emerging threats.
The expanded dataset maintains comparable average attack success rates to the original human-crafted prompts, meaning the AI-generated prompts are just as effective at uncovering model vulnerabilities. Importantly, Seed2Harvest dramatically increases prompt diversity, featuring 535 unique geographic locations and a higher Shannon entropy, compared to 58 locations in the original dataset. This enhanced diversity is crucial for comprehensive safety evaluation, as it helps identify regionally-specific vulnerabilities and cultural biases that might otherwise go unnoticed.
Also Read:
- Enhancing Safety in Text-to-Image AI with Distributional Red Teaming
- Unmasking Hidden Memorization in Text-to-Image AI
Challenges and Future Directions
While powerful, the research highlights that the quality and breadth of the initial human seed prompts are critical. If the seed set lacks variety, the expansions might inadvertently amplify existing biases. Additionally, the reliance on LLMs, which are often primarily trained on English texts, can introduce linguistic and cultural knowledge gaps, potentially leading to stereotypes or oversimplifications despite efforts to include diverse geographical cues.
The researchers also acknowledge the inherent risks of creating large datasets of adversarial prompts, as they could potentially be misused. Therefore, they have chosen not to broadly share their newly generated datasets to discourage malicious use.
In conclusion, Seed2Harvest represents a significant step forward in T2I model safety evaluation. By intelligently combining human insight with AI’s capacity for scale, it offers a more consistent, diverse, and efficient approach to red-teaming, paving the way for safer and more equitable generative AI models.
