TLDR: A new study evaluates how accurately Text-to-Image models like Stable Diffusion and Janus Pro adhere to simple text prompts, especially concerning object shape and location. Researchers found these models struggle significantly with precise spatial positioning, even for basic geometric shapes, highlighting a need for architectural improvements rather than just fine-tuning.
In the rapidly evolving landscape of artificial intelligence, Text-to-Image models have emerged as powerful tools capable of synthesizing high-fidelity visuals from natural language descriptions. These models, such as Stable Diffusion and Janus Pro, have pushed the boundaries of imaginative image generation. However, a recent research paper delves into a critical aspect of their performance: how robustly they adhere to the instructions given in text prompts.
The paper, titled “TOWARDS EVALUATING ROBUSTNESS OF PROMPT ADHERENCE IN TEXT TO IMAGE MODELS,” highlights a significant gap in current research concerning the reliability and effectiveness of multimodal Large Language Models (LLMs) and Text-to-Image models. While text-only LLMs have seen extensive research into their reliability, their multimodal counterparts are still relatively new, and their robustness remains constrained due to insufficient evaluation.
A Novel Approach to Evaluation
To address this, researchers Sujith Vemishetty, Advitiya Arora, and Anupama Sharma developed a comprehensive evaluation framework. Their primary focus was on assessing how well Text-to-Image models generate images that conform to specified factors of variation in input text prompts. They created a novel dataset, inspired by the well-known dSprites dataset, but with crucial improvements. This new dataset features 256×256 resolution images of simple geometric shapes—squares, circles, and triangles—at various scales and precise X and Y positions. This design overcomes limitations of older datasets, such as low resolution and ambiguous shapes, allowing for a clearer assessment of prompt adherence.
The evaluation methodology is innovative, combining both image understanding and image generation models in an automated pipeline. Initially, the gpt-4o model was used to generate detailed text descriptions for ground-truth images, including predictions of the shape present and its quadrant location. These descriptions were then fed into various Text-to-Image models, including Stable Diffusion 3 Medium, Stable Diffusion 3.5 Large, Stable Diffusion 3.5 Large Turbo, Janus Pro 1B, and Janus Pro 7B, to generate new images. This process was iterated, with gpt-4o re-describing the generated images, and those descriptions then used to generate images again, creating a feedback loop to observe performance degradation.
Key Findings: Challenges in Prompt Adherence
The study measured three key metrics: reconstruction loss (how well generated images could be reconstructed by Variational Autoencoders trained on the dataset), and the degradation of F1-Scores for both shape and quadrant predictions made by gpt-4o on the generated images. The results revealed several important insights:
- Resolution Matters: For Stable Diffusion models, prompt adherence significantly decreased at lower resolutions. Images generated at 1024×1024 resolution showed the highest F1-Scores for shape prediction, while 256×256 resolution images performed poorly, indicating a struggle to adhere to simple geometric shapes in straightforward binary images at lower resolutions.
- Spatial Awareness is Lacking: Predicting the quadrant where a shape should be located proved to be substantially more challenging for all models. The highest F1-Score for quadrant prediction was very low, demonstrating that these models struggle significantly with understanding and accurately placing shapes at a given location, even with a high guidance scale configured to enforce strict prompt adherence.
- Performance Degradation Over Iterations: Across both Stable Diffusion and Janus Pro models, F1-Scores for shape and quadrant prediction, as well as reconstruction quality, declined significantly over subsequent iterations of image generation and re-description. This indicates a loss of information and faithfulness as the process progresses.
- Janus Pro vs. Stable Diffusion: Interestingly, Janus Pro models generally outperformed Stable Diffusion variants in terms of prompt adherence and reconstruction loss, despite being limited to a lower resolution (384×384). However, one caveat noted was that Stable Diffusion 3 Medium, while showing lower reconstruction loss, often generated blank images, which artificially skewed its scores.
The researchers emphasize that while models like Stable Diffusion and Janus Pro can match the shape given in a text description reasonably well, their ability to position these shapes accurately within an image is severely inadequate. This suggests a fundamental limitation in their spatial awareness.
Also Read:
- Mapping the Past: AI Uses Visuals to Pinpoint Historical Locations
- Advancing Multimodal AI: A New Model for Unified General and Spatial Understanding
Future Directions
The paper concludes that current Text-to-Image models struggle with rigid prompt adherence, particularly for simple binary images with precise spatial requirements. The authors suggest that simple fine-tuning is not a scalable solution for the infinite concepts in the real world. Instead, a deeper analysis of the internal architecture of these models is needed, with architectural modifications in the pre-training phase to foster a better conceptual understanding and improve prompt adherence. For more details, you can read the full research paper here.
