TLDR: GenTune is a human-centered AI system designed for environment designers that addresses challenges in refining AI-generated images. It introduces “traceable prompts” allowing designers to link image elements to specific prompt labels, and “semantic-guided refinement” for precise, context-aware edits. Studies show GenTune significantly improves prompt-image understanding, refinement quality, efficiency, and user satisfaction, leading to its adoption in professional design studios.
Environment designers in the entertainment industry are constantly crafting imaginative 2D and 3D scenes for games, films, and television. This work demands both precise control over small details and a consistent look across the entire scene. With the rise of generative AI (GenAI) tools, designers have started integrating them into their workflows. However, this integration hasn’t been without its challenges.
A recent study with 10 designers highlighted two main issues: first, the lengthy prompts generated by large language models (LLMs) make it hard to pinpoint which keywords control specific visual elements. This leads to a lot of guesswork when trying to make revisions. Second, while inpainting (a technique for localized edits) is useful, it often struggles to maintain global consistency and correctness, leading to mismatched lighting, styles, or historical inaccuracies.
To address these critical challenges, researchers have developed GenTune, a new approach designed to enhance human-AI collaboration by making the connection between AI-generated prompts and image content much clearer. GenTune allows designers to select any element in a generated image, trace it back to the specific prompt labels that created it, and then revise those labels to guide precise yet globally consistent image refinement.
How GenTune Works
GenTune is built on two core modules:
1. Traceable Prompt: When a designer starts with a simple input, like “Design a European 1930s Urban Street Scene,” a brainstorming LLM expands it into a detailed prompt for image generation. A separate label extraction LLM then identifies key elements from this prompt. In GenTune’s interface, designers can select an area of interest in the generated image (e.g., “Vintage Cars”). The system then reveals the corresponding prompt label and highlights the relevant section in the structured prompt panel. This transparency helps designers understand exactly how different parts of the prompt influence the visual elements.
2. Semantic-Guided Refinement: Once a label is traced, designers can refine the image using natural language instructions or by providing a reference image. GenTune offers three refinement modes:
- Global Refinement: For broad, whole-image edits like changing the overall style, mood, or lighting without needing to select specific regions.
- Semantic-Guided Prompt Refinement with Controlled Seed: This innovative method makes targeted edits to the original prompt based on the user’s input and the selected label. The image is then regenerated using the same ‘seed’ (a unique identifier that helps maintain consistency). This ensures that changes apply only to the intended element while preserving the overall coherence of the image. For example, if “Vintage Cars” are selected and the instruction is to “Replace with Vintage Electrical Tram,” GenTune replaces the cars and even adds overhead wires for contextual consistency, keeping the rest of the scene largely unchanged.
- Semantic-Guided Inpainting: For more localized edits, GenTune accepts simple natural language commands (e.g., “add some merchants”) and generates context-aware prompts. This ensures that the inserted content remains stylistically and semantically consistent with the rest of the scene.
GenTune supports a progressive refinement workflow, allowing designers to start with global adjustments and then move to fine-grained control. It also provides helpful refinement suggestions, both global and label-specific, to inspire new ideas.
Real-World Impact and Studies
The development of GenTune began with a formative study involving 10 designers, which helped identify the core challenges. Following this, a summative study was conducted with 20 designers (15 professionals, 5 students) experienced with GenAI tools. Participants compared GenTune to a baseline system that mimicked current industry practices.
The results were overwhelmingly positive: GenTune significantly improved prompt-image comprehension, refinement quality, and efficiency. Designers reported better understanding of how prompts related to image elements, more effective and accurate refinements, and higher overall satisfaction. They also found GenTune to be more controllable and aligned with their expectations, reducing the need for time-consuming trial-and-error.
A follow-up field study with two design studios further demonstrated GenTune’s effectiveness in real-world professional settings. Designers integrated GenTune into their ongoing commercial projects for three days and reported improved efficiency, higher output quality, and a significant reduction in time spent communicating design intent to clients and directors. Both studios have continued to use GenTune in their commercial projects, producing over 40 environments to date, showcasing its strong potential for adoption.
Also Read:
- Intelligent Configuration Optimization with Language-Guided Tuning
- LENS: Unifying Language Understanding and Pixel Segmentation
Looking Ahead
GenTune represents a significant step forward in human-centered AI, offering a model-agnostic paradigm for traceable, element-level control in creative workflows. By making the generative process more transparent and controllable, GenTune empowers designers to focus on creative decisions, boosting their trust in AI tools and enhancing their ability to explore new visual possibilities. This approach has broad implications for other visual domains, such as character design, interior design, and even video and animation generation.
For more detailed information, you can read the full research paper: GenTune: Toward Traceable Prompts to Improve Controllability of Image Refinement in Environment Design.
