TLDR: The paper introduces a lightweight, plug-and-play method to enhance spatial relationships in text-to-image generation. It uses a fine-tuned T5-small language model to convert natural language prompts into tuple-based structured information, which is then appended to the original prompt and fed into diffusion models like Stable Diffusion XL. This approach significantly improves spatial accuracy, color, and shape alignment without compromising image quality, outperforming existing methods and demonstrating that automatically generated tuples are comparable to human-crafted ones.
Text-to-image (T2I) generation has seen remarkable progress with models like Stable Diffusion and DALL-E 3, allowing the creation of highly realistic and imaginative images from simple text descriptions. However, a persistent challenge remains: accurately representing spatial relationships described in natural language prompts. For instance, if you ask for “a red ball on top of a blue box,” current models might struggle to place the objects correctly or consistently.
Previous attempts to address this issue have involved complex methods like prompt optimization, spatially grounded generation, and semantic refinement. While these approaches offer some improvements, they often come with significant computational costs and can be difficult to generalize across different scenarios due to their intricate designs.
A new research paper titled “STRUCTURED INFORMATION FOR IMPROVING SPATIAL RELATIONSHIPS IN TEXT-TO-IMAGE GENERATION” by Sander Schildermans, Chang Tian, Ying Jiao, and Marie-Francine Moens introduces a lightweight and practical solution. This innovative method enhances prompts with structured, tuple-based information, making it easier for T2I models to understand and render spatial arrangements accurately.
The StructuredPrompter Approach
The core of this new approach, dubbed StructuredPrompter, involves a fine-tuned T5-small language model. This model automatically converts natural language text into a structured format consisting of object tuples (e.g., (color, shape)) and spatial relation tuples (e.g., (subjectID, relation, objectID)). For example, a prompt like “Add a cyan cube at the center. Add a cyan cylinder on the right in front of it” would be transformed into tuples such as (cyan, cube), (cyan, cylinder), (cyanCylinder, rightOf, cyanCube), and (cyanCylinder, frontOf, cyanCube).
This structured information is then simply appended to the original natural language prompt and fed directly into powerful T2I models like Stable Diffusion XL. Crucially, this process requires no modifications to the underlying diffusion model’s architecture or parameters, making it a “plug-and-play” solution that adds minimal computational overhead.
Key Contributions and Benefits
The researchers highlight several significant contributions:
- A novel tuple-based representation that explicitly encodes objects and their spatial relations in a compact, model-friendly format.
- A lightweight and portable converter built on T5-small, a language model with only 60 million parameters, requiring limited training data.
- Demonstrated improvements in handling spatial relationships, along with higher image quality as measured by the Inception Score. The method outperforms plain prompt inputs and leading baselines like RealCompo and DPT-T2I.
- A plug-and-play generation pipeline that enhances spatial alignment while remaining portable and easy to integrate into existing T2I workflows.
Evaluation and Results
To evaluate the effectiveness of StructuredPrompter, the researchers used the TV-Logic dataset, which focuses on relational text descriptions of simple colored shapes. They fine-tuned the T5-small model using 500 training prompts, with ground-truth structured information generated by GPT-4o.
The generated images were assessed using Qwen2.5-VL-3B-Instruct, an AI judge, to verify the correctness of spatial relationships, color, and shape alignment. The results showed that augmenting prompts with structured information substantially improved the modeling of spatial relationships, as well as color and shape alignment. For instance, the method achieved a spatial accuracy of 0.473 compared to 0.446 for plain prompts in SDXL, and significantly outperformed baselines like RealCompo (0.310 spatial accuracy).
Furthermore, the overall visual quality of the generated images was evaluated using the Inception Score (IS). StructuredPrompter achieved higher IS values (6.94) than plain prompts (6.71) and even outperformed or matched leading baselines, confirming that the improvements in spatial accuracy do not come at the expense of image quality.
Interestingly, the study also found that automatically generated tuples from the fine-tuned T5-small model achieved results comparable to, and even slightly better than, manually created tuples, demonstrating the reliability and efficiency of the lightweight language model for this task.
Also Read:
- Cognitive-Inspired AI: A New Method for Attention Management in Transformers
- PicLumen AI Unveils Advanced Generative Art Platform for Creative Industries in 2025
Conclusion
This research presents a practical and effective method for enhancing spatial faithfulness in text-to-image generation. By leveraging a fine-tuned T5-small model to append structured information to plain text prompts, the StructuredPrompter framework achieves measurable improvements in spatial accuracy and maintains high visual quality. Its plug-and-play nature, minimal computational overhead, and strong performance make it a promising solution for addressing a key limitation in current large-scale generative systems. You can read the full research paper here: Structured Information for Improving Spatial Relationships in Text-to-Image Generation.
