TLDR: SMARLI is a novel framework for layout-to-image generation using autoregressive models. It integrates spatial layout constraints through a structured masking strategy that manages attention between global text prompts, layout tokens, and image tokens, preventing feature entanglement and ensuring accurate spatial control. Additionally, it employs a GRPO-based post-training scheme with a dedicated layout reward and an image quality reward to enhance generation quality and layout accuracy. Experimental results show SMARLI achieves superior layout control compared to diffusion-based counterparts.
Autoregressive (AR) models have shown great promise in generating images, similar to their success in large language models. However, applying them to generate images based on specific layouts – where objects need to be placed at particular spatial locations with defined characteristics – has been a significant challenge. This is primarily due to the sparse nature of layout information and the risk of different features getting mixed up during the generation process.
A new framework called Structured Masking for AR-based Layout-to-Image (SMARLI) has been introduced to tackle these challenges. SMARLI is designed to seamlessly integrate spatial layout constraints into AR-based image generation, offering a novel approach to creating images with precise control over object placement and attributes.
The core of SMARLI’s innovation lies in two main components: a specially designed structured masking strategy and a Group Relative Policy Optimization (GRPO)-based post-training scheme.
Structured Masking for Precise Control
To give AR models better control over layouts, SMARLI employs a structured masking strategy during the attention computation within the transformer architecture. This strategy carefully governs how different types of tokens – global text prompts, layout information, and image tokens – interact with each other. Layout tokens are allowed to draw context from both the overall text prompt and local layout details within their specific region, while being masked from other regions to prevent confusion. Image tokens, in turn, receive guidance from both the global text prompt and the layout tokens relevant to their specific area. This ensures that the image generation process is guided by the correct context, avoiding attribute mix-ups and maintaining precise alignment.
This intelligent masking introduces a structured inductive prior that aligns with how prompts, layouts, and images are organized sequentially. This makes training more efficient without needing major changes to the underlying AR model architecture.
Enhancing Quality with GRPO-based Post-training
To further boost the quality of generated images and the accuracy of layout adherence, SMARLI incorporates a GRPO-based post-training scheme. This scheme is specifically adapted for next-set-based AR models, which predict groups of image tokens at each step rather than one by one, improving efficiency for high-resolution images.
During this post-training phase, two types of rewards are used: a human preference score (HPSv2.1) to improve overall image quality, and a novel layout reward. The layout reward is crucial; it calculates the CLIP similarity between cropped regions of the generated image and their corresponding textual descriptions from the layout. This ensures that the model not only generates the correct objects but also accurately renders their specified colors, textures, and shapes within their designated areas. By combining these rewards, SMARLI can generate images with improved visual quality while achieving better layout alignment.
Also Read:
- Boosting Efficiency in UI Code Generation with Smart Token Compression
- Understanding How AI Connects Words to Images
Experimental Results and Impact
SMARLI was evaluated on the LayoutSAM-Eval benchmark, a dataset designed for assessing layout-to-image generation. The framework demonstrated strong performance in layout control, excelling in spatial positioning, color, texture, and shape accuracy. It also achieved competitive image quality, with favorable FID scores compared to state-of-the-art diffusion-based models like GLIGEN, InstanceDiffusion, and SiamLayout.
Ablation studies confirmed the effectiveness of both the structured masking strategy and the GRPO-based post-training. Removing either component led to a significant degradation in performance, highlighting their critical roles in SMARLI’s success.
This research marks a significant step forward in autoregressive text-to-image generation, demonstrating that AR models can be powerful competitors to diffusion models in controllable image synthesis. For more details, you can read the full paper here.
