TLDR: A new research paper introduces Re-LAION-Caption 19M, a dataset of 19 million high-quality images with structured captions (subject, setting, aesthetics, camera details) generated by LLaVA-Next. By training text-to-image models like PixArt-Σ and Stable Diffusion 2 on this structured data, the researchers demonstrate significant improvements in prompt adherence and text-image alignment compared to models trained on unstructured or shuffled captions. The work suggests that consistent captioning simplifies model learning and enhances controllability.
Generative text-to-image models, while powerful, often face a significant challenge: accurately adhering to user prompts. This issue largely stems from the vast, yet often noisy and unstructured, datasets they are trained on, such as LAION-5B. This forces users to engage in extensive “prompt engineering” to achieve their desired visual outputs.
A new research paper, “Structured Captions Improve Prompt Adherence in Text-to-Image Models (Re-LAION-Caption 19M),” proposes an innovative solution: enforcing a consistent caption structure during model training. The authors, Nicholas Merchant, Haitz S´aez de Oc´ariz Borde, Andrei Cristian Popescu, and Carlos Garcia Jurado Suarez, argue that this structured approach can dramatically enhance model controllability and alignment with user intentions.
To demonstrate this, they introduce Re-LAION-Caption 19M, a high-quality subset derived from Re-LAION-5B. This dataset comprises 19 million images, each measuring 1024×1024 pixels, paired with meticulously generated captions. These captions are not random; they follow a precise four-part template: subject, setting, aesthetics, and camera details. The captions themselves were created using a Mistral 7B Instruct-based LLaVA-Next model.
The core hypothesis is that this consistent structure simplifies the learning process for text-to-image models. Instead of having to learn to be invariant to countless different caption formats, the model can focus its capacity on understanding the semantic content. This also benefits users, as their prompts can more easily align with the model’s training distribution, reducing the need for complex prompt engineering at the inference stage.
The researchers fine-tuned popular text-to-image models, PixArt-Σ and Stable Diffusion 2, using two versions of their dataset: one with the structured captions and another with randomly shuffled captions. Their experiments consistently showed that models trained with structured captions achieved higher text-image alignment scores, as evaluated by visual question answering (VQA) models like LLaVA-based and InstructBLIP-based systems. This indicates that the structured approach indeed leads to better prompt adherence.
Also Read:
- Benchmarking Vision-Language Models: Unpacking Performance Across Diverse AI Tasks
- Efficient Image Creation: A New Approach to Autoregressive Models
This work highlights a crucial aspect of generative AI development: the quality and organization of training data are paramount for model reliability and predictability. By open-sourcing Re-LAION-Caption 19M, the team aims to foster further advancements in creating more controllable and user-friendly generative AI systems. You can find the full research paper here: Structured Captions Improve Prompt Adherence in Text-to-Image Models.
