TLDR: Researchers introduce a novel 3D gaze redirection framework that uses an explicit 3D eyeball structure with 3D Gaussian Splatting to generate photorealistic images with accurate and controllable gaze. The method allows for explicit eyeball rotation and includes an adaptive deformation module for subtle muscle movements around the eyes, enabling independent eye control and outperforming previous state-of-the-art techniques in image quality and gaze accuracy.
A new research paper titled “Roll Your Eyes: Gaze Redirection via Explicit 3D Eyeball Rotation” introduces a groundbreaking approach to manipulating eye gaze in digital images, promising more realistic and controllable virtual interactions. This work, by YoungChan Choi, HengFei Wang, YiHua Cheng, Boeun Kim, Hyung Jin Chang, YoungGeun Choi, and Sang-Il Choi, addresses a significant challenge in computer vision and graphics: creating digital avatars that can express natural and accurate gaze.
Traditional methods for gaze redirection often rely on neural radiance fields (NeRFs), which use implicit neural representations. While effective, these approaches struggle with explicitly modeling the 3D structure of the eyeball, making precise rotation and translation difficult. This limitation can lead to less realistic eye movements in digital representations.
The core innovation of this new framework is its use of an explicit 3D eyeball structure, integrated with 3D Gaussian Splatting (3DGS). Unlike implicit methods, 3DGS provides a clear 3D representation, making it ideal for modeling the eyeball. By physically rotating and translating this dedicated 3D eyeball structure, the system can generate photorealistic images that accurately reflect a desired gaze direction.
Beyond just rotating the eyeball, the researchers also developed an adaptive deformation module. This module is crucial for replicating the subtle muscle movements around the eyes that naturally occur when a person changes their gaze. This attention to detail significantly enhances the realism of the generated faces, capturing nuances that are often missed by other methods.
A notable feature of this framework is its ability to control the left and right eyes independently. This opens up possibilities for synthesizing complex and even unconventional gaze patterns, such as cross-eyed looks, which are challenging to capture or simulate in real-world scenarios. This fine-grained control expands the creative potential for gaze synthesis in various applications.
The effectiveness of the framework was demonstrated through experiments on the ETH-XGaze dataset. The results show that the proposed method outperforms existing state-of-the-art techniques in terms of image quality and gaze estimation accuracy. It also maintains strong identity preservation, meaning the generated images look like the original subject, just with a different gaze. The system is also built upon 3DGS, which allows for higher detail preservation and faster rendering speeds compared to NeRF-based approaches.
Ablation studies confirmed the importance of the newly introduced loss functions and the gaze-guided deformation field. These components were shown to be critical for ensuring that the eyeball and facial representations are disentangled and that subtle muscle movements around the eyes are accurately captured, preventing visual distortions and improving overall realism.
Also Read:
- UW-3DGS: Advancing Underwater 3D Scene Reconstruction with Physics-Aware Gaussian Splatting
- SIFThinker: Bringing Deeper Spatial Awareness to AI Vision
This novel gaze redirection framework marks a significant step forward in creating more lifelike and expressive digital avatars. By enabling precise and independent control over eye movements and capturing the subtle surrounding muscle deformations, this technology has vast potential for applications in virtual reality, augmented reality, remote conferencing, and the development of highly immersive digital avatars. You can read the full research paper here.
