TLDR: MEPG (Multi-Expert Planning and Generation) is a novel framework for text-to-image generation that addresses the challenges of complex, multi-element prompts and limited stylistic diversity. It integrates a Position-Style-Aware (PSA) module, which uses a fine-tuned LLM to decompose prompts into precise spatial coordinates and style instructions, with a Multi-Expert Diffusion (MED) module that dynamically routes specialized expert models for cross-region generation. This approach allows for fine-grained control over object placement, maintains global aesthetic quality, and supports diverse artistic styles, significantly outperforming existing methods.
Text-to-image diffusion models have made incredible strides in generating high-quality images from simple descriptions. However, they often face significant challenges when confronted with complex prompts involving multiple elements, intricate relationships, or a demand for diverse artistic styles. Imagine trying to describe a bustling city scene with specific objects in precise locations and varying artistic treatments – current models often fall short in these nuanced scenarios.
Addressing these limitations, researchers Zhao Yuan and Lin Liu have introduced a novel framework called MEPG: Multi-Expert Planning and Generation for Compositionally-Rich Image Generation. This innovative approach integrates the reasoning capabilities of large language models (LLMs) with specialized image generation modules, offering a more precise and versatile way to create images.
The MEPG framework is built upon two core components:
Position-Style-Aware (PSA) Module
This module acts as the intelligent planner. It utilizes a supervised fine-tuned large language model (LLM) to meticulously break down an input text prompt. Instead of just understanding the words, the PSA module decomposes the prompt into precise spatial coordinates for each element and encodes specific style instructions. For instance, if you ask for a ‘red car on the left and a blue house on the right in a watercolor style,’ the PSA module will determine the exact bounding boxes for the car and house and assign the ‘watercolor’ style to the relevant regions. This process is enhanced by a Low-Rank Adaptation (LoRA) model, which helps the LLM achieve greater accuracy in spatial reasoning, a common weakness in standard LLMs.
Also Read:
- Precision in Pixels: FocusDPO Improves Personalized Multi-Subject Image Creation
- Enhancing Visual Question Answering Through Iterative Multimodal Retrieval
Multi-Expert Diffusion (MED) Module
Once the PSA module has created a detailed plan, the MED module takes over the generation process. Unlike single-model architectures that struggle to balance diverse requirements, MED employs a dynamic expert routing system. It activates specialized models, or ‘experts,’ for different local regions and global areas of the image. For example, a ‘realism expert’ might be engaged for a photographic element, while a ‘stylization specialist’ could be used for an artistic background. This dynamic selection, managed by an attention-based gating mechanism, allows for cross-region generation, ensuring both local detail and global coherence. The modular design of MED also means that expert models can be easily integrated or replaced, making the framework highly extensible.
MEPG also incorporates a ‘cross denoising scheduling’ mechanism. This clever approach balances global consistency with local detail control. During the image generation process, it dedicates an initial phase to local denoising, where individual experts refine their assigned regions. Subsequently, a global denoising phase ensures that all these localized efforts come together harmoniously, maintaining the overall structural alignment and aesthetic quality of the image.
The framework’s effectiveness has been rigorously tested through experiments, demonstrating significant improvements over baseline models in both image quality and style diversity, particularly when handling complex, multi-element prompts. An interactive interface further empowers users by allowing real-time spatial layout editing and per-region style selection from a portfolio of experts, offering unprecedented control over the creative process.
In essence, MEPG transforms image generation from a direct process into a sophisticated workflow involving reasoning, layout planning, and style orchestration. By integrating LLM-based spatial planning with a multi-expert collaborative generation system, MEPG offers a systematic and powerful solution for creating compositionally rich and stylistically diverse images. You can read the full research paper here.
