TLDR: TransLight is a new AI framework that allows users to transfer specific light effects from a reference image to a target image with high fidelity and control. It achieves this by first “decoupling” or separating the light effect from the reference image’s content using fine-tuned diffusion models, then applying it to the target image while preserving its original content. Users can adjust the position, direction, and intensity of the transferred light.
Imagine being able to take a stunning light effect from one photograph – perhaps a beautiful sunbeam, a dramatic lens flare, or a unique color cast – and seamlessly apply it to another one, all while keeping the original content of your target image perfectly intact. This challenging task, known as light effect transfer, has largely been out of reach for existing image editing and relighting methods, which often struggle to provide customized control without altering the image’s core content.
A new framework called TransLight aims to solve this problem. Developed by researchers from Huazhong University of Science and Technology and vivo Mobile Communication Co., Ltd., TransLight introduces a novel approach for high-fidelity and high-freedom transfer of light effects between images. The core innovation lies in its ability to accurately separate light effects from image content, a process they call Generative Decoupling.
Generative Decoupling: The Key to Separation
The most critical and challenging step in TransLight is extracting the light effect from a reference image without bringing along any unwanted content. To achieve this, the researchers developed Generative Decoupling, which utilizes two fine-tuned diffusion models. These models are trained to precisely separate image content from light effects. This process also involved creating a massive new dataset of over a million image–content–light triplets, which are essential for training the system.
One diffusion model, the ‘light removal model,’ learns to eliminate prominent light effects like lens flares or Tyndall effects from natural images, leaving behind only the content. The other, the ‘light extraction model,’ is designed to isolate just the light effects, ensuring no other objects or content are retained. This careful disentanglement is what gives TransLight its unique flexibility and control.
How TransLight Works
After the generative decoupling process, TransLight employs a two-stage training approach. It uses IC-Light, a powerful generative model, as its foundation. In the first stage, the model is fine-tuned to ensure it preserves the background and maintains content consistency when light effects are added. The second stage involves training an additional ControlNet, which is responsible for injecting the extracted light effects into the target image.
During inference, the light extraction model first pulls the desired light effect from a reference image. This extracted light effect is then fed into the ControlNet. What’s remarkable is the level of control TransLight offers: users can freely adjust the position (horizontal or vertical translation), direction (flipping or rotating), and even the intensity of the transferred light effect on the target image. This allows for highly customized and natural integration of lighting.
Also Read:
- RefAdGen: Advancing AI-Powered Advertising Image Creation
- GaitCrafter: AI Model Synthesizes Realistic Walking Patterns to Advance Biometric Recognition
Performance and Impact
Experimental results show that TransLight significantly outperforms existing state-of-the-art illumination editing methods like IC-Light, SeedEdit3.0, and GPT-Image-1 in transferring light effects. While other methods might introduce content from the reference image, alter character appearance, or fail to understand the geometric structure of light effects, TransLight consistently delivers visually authentic results. It achieves a superior Light FID score, a metric used to evaluate lighting transfer capability, demonstrating its ability to generate images that are closer to real-world data.
This breakthrough opens up new possibilities for personalized image editing. For instance, a photographer could capture a portrait and then enhance it with artistic light effects sourced from high-quality templates or other images, all while preserving the original subject. The framework provides a robust and flexible foundation for future advancements in image-guided light effect transfer, promising significant potential for practical lighting editing tasks.
For more technical details, you can read the full research paper: TransLight: Image-Guided Customized Lighting Control with Generative Decoupling.
