TLDR: ReFocus is a training-free framework that improves text-to-image models’ ability to handle complex compositions. It uses Large Language Models (LLMs) to automatically generate explicit layouts from text prompts, then iteratively refines and re-ranks generated images based on both scene-level and object-level evaluations. This process leads to more accurate object counts, attributes, and spatial relations in generated images without requiring additional training.
The field of text-to-image models has seen remarkable advancements, creating incredibly realistic and diverse images from simple text prompts. However, a persistent challenge remains: compositionality. These models often struggle with accurately rendering object counts, attributes, and spatial relationships. For instance, a prompt asking for “four giraffes” might produce the wrong number, or “a chair left of a zebra” could result in incorrect spatial placement. This gap between user intent and model output highlights a significant limitation.
Existing solutions often require users to provide both a text prompt and a detailed layout, such as bounding boxes, which can be cumbersome. Additionally, the inherent randomness of diffusion models can lead to inconsistent image quality, forcing users to generate multiple samples to get a satisfactory result.
Introducing ReFocus: A New Approach
A new framework called ReFocus addresses these issues by combining an object-centric approach with self-refinement, all without requiring additional training. ReFocus aims to improve how well generated images match the specified layout while maintaining high aesthetic quality.
The framework works in three main phases:
First, LLM-based Layout Generation: Instead of manual input, ReFocus uses large language models (LLMs) to automatically create explicit layouts from the user’s text prompt. The LLM acts as a “layout planner,” proposing a plausible and balanced arrangement for all objects mentioned in the caption. To prevent issues like heavy object overlap, the system applies slight adjustments to the generated layout boxes.
Second, Layout-Grounding Initial Generation: With the prompt and the LLM-generated layout, a layout-conditioned diffusion model generates an initial set of draft images. Unlike standard text-to-image models that might misplace objects, this phase uses the layout to establish a basic compositional structure from the start, focusing on individual objects. Importantly, these drafts are not the final output but serve as a foundation for further refinement.
Third, Iterative Self-Refinement: The initial drafts, while structurally sound, often lack photorealistic detail. This phase iteratively refines and validates the images. It uses a “preference re-ranking” module that evaluates candidates based on both scene-level (overall prompt alignment) and object-level (individual object presence and identity) judgments. This hybrid scoring system provides a more accurate assessment than just scene-level evaluation. The top-ranked images are then refined using a lightweight model, enhancing details like texture and lighting without altering the established geometric arrangement. This re-ranking and refinement process is repeated in a loop, progressively improving both overall scene alignment and fine-grained object fidelity. This iterative approach allows the model to correct errors like missing objects or implausible details.
Also Read:
- Enhancing Image Descriptions with Hierarchical Planning
- GRAID: Generating High-Quality Data to Enhance Spatial Reasoning in Vision Language Models
Key Advantages of ReFocus
ReFocus stands out because it integrates several key features: it’s object-centric, uses inference-time scaling, incorporates self-refinement, and is entirely training-free. Unlike other methods that might use inference-time scaling but lack refinement, or those that refine during inference but require additional training, ReFocus combines all these benefits.
The framework has been tested on challenging benchmarks like GenEval and HPS v2.1, showing superior performance in compositional accuracy, especially for object counting, positioning, and attribute rendering. For example, it significantly improves position accuracy compared to other models. Visually, ReFocus consistently produces images that accurately reflect prompt details, such as “a purple elephant and a brown sports ball” or “four traffic signs,” correctly rendering object counts, colors, and relative positions where other methods often fail.
The authors have made the code available at their GitHub repository. You can find more details about this innovative approach in the full research paper: COMPOSITIONAL IMAGE SYNTHESIS WITH INFERENCE-TIME SCALING.
