TLDR: RefAdGen is a new AI framework for generating high-fidelity advertising images from product photos and text descriptions without extensive fine-tuning. It introduces AdProd-100K, a large dataset with a dual augmentation strategy for 3D-aware product representation. RefAdGen uses a decoupled dual U-Net architecture and an Attention Fusion Module to precisely control spatial layout and integrate product identity, achieving state-of-the-art results in realism and fidelity, even with real-world images.
The world of digital marketing and e-commerce is constantly seeking innovative ways to create visually engaging advertising content quickly and cost-effectively. Traditional methods, relying on expensive photoshoots and manual design, often create bottlenecks in content creation. While Artificial Intelligence Generated Content (AIGC), particularly diffusion models, offers a promising solution, a significant challenge has persisted: how to generate high-fidelity advertising images that accurately preserve the visual characteristics of a given product while also creating compelling scenes based on user descriptions.
Existing AIGC techniques often face a dilemma between visual fidelity and computational efficiency. Methods that achieve high fidelity, like DreamBooth, require extensive fine-tuning for each product, making them impractical for platforms with vast product catalogs due to high training and storage costs. On the other hand, tuning-free methods, while efficient and scalable, frequently struggle to maintain crucial product details such as unique textures, shapes, and logos, which are vital for brand identity in advertising.
Introducing RefAdGen: Bridging the Fidelity-Efficiency Gap
To address this critical limitation, researchers have introduced RefAdGen, a novel generation framework designed to produce high-fidelity advertising images without the need for extensive fine-tuning for each product. This innovative approach tackles the fidelity-efficiency dilemma head-on, offering a scalable and cost-effective alternative to traditional marketing workflows. You can find the full research paper here: RefAdGen: High-Fidelity Advertising Image Generation.
AdProd-100K: A New Benchmark for Advertising Image Generation
A key enabler for RefAdGen is the creation of AdProd-100K, a large-scale dataset specifically constructed for reference-based advertising image generation. This dataset comprises 100,000 high-quality triplets, each containing a textual scene description, a product image, and an advertising image. What makes AdProd-100K unique is its dual data augmentation strategy. This strategy combines multi-view synthesis, which renders novel yet similar views of products using 3D Gaussian Splatting, and image degradation, which simulates real-world imaging variations like noise and shadows. This dual approach encourages models to learn robust, 3D-aware representations of products rather than simply “copy-pasting” 2D images, leading to more realistic and generalizable results.
How RefAdGen Works: A Decoupled Design
RefAdGen’s core innovation lies in its decoupled generation architecture, which separates the complex synthesis process into two distinct tasks: spatial layout control and identity feature fusion. This is achieved through a dual U-Net backbone and an Attention Fusion Module (AFM).
- Dual U-Net Architecture: This architecture uses two U-Nets derived from the same pre-trained model. The “Reference U-Net” acts as a dedicated identity feature extractor, learning to extract critical features from the input product image. The “Generation U-Net” is the primary network for synthesizing the final advertising images, guided by the extracted product features and the user-provided scene description. A crucial aspect is that the Generation U-Net’s input is modified to accept a product mask, providing precise spatial guidance early in the process.
- Attention Fusion Module (AFM): This module efficiently injects identity features from the Reference U-Net into the Generation U-Net. It combines self-attention for scene structure with cross-attention for identity injection, ensuring that product details are faithfully preserved within the generated scene. By injecting the mask at the U-Net input, the model learns to apply identity features only within the intended foreground, preventing feature conflicts and maintaining spatial coherence.
Also Read:
- Crafting Images with Layered Detail: Introducing Next Visual Granularity Generation
- MOON: A New Generative AI Model for Deeper E-commerce Product Understanding
State-of-the-Art Performance and Real-World Applicability
Extensive experiments demonstrate that RefAdGen achieves state-of-the-art performance across various metrics, including CLIP-Score for text adherence, FID for realism, and MP-LPIPS/LPIPS for perceptual similarity and identity preservation. It consistently outperforms existing methods like IP-Adapter, ControlNet, and Instruct-Pix2Pix. The framework shows robust generalization capabilities, maintaining high fidelity for both unseen products and challenging real-world “in-the-wild” images captured with mobile phones. A user study further confirmed that images generated by RefAdGen are not only highly realistic but also aesthetically preferred by users, highlighting their strong commercial appeal.
RefAdGen represents a significant step forward in AIGC for advertising, offering a powerful, scalable, and cost-effective solution for generating high-fidelity advertising images. The public availability of its code and datasets will undoubtedly foster further research and development in this crucial commercial domain.
