TLDR: EAvatar is a novel 3D head avatar reconstruction framework that utilizes dynamic 3D Gaussian Splatting to create highly realistic and controllable digital faces. It addresses limitations in capturing fine-grained facial expressions and local texture continuity by introducing a sparse expression control mechanism, a deformation-aware Gaussian splitting strategy, and a structure-aware geometry modeling module guided by generative AI priors. The method achieves superior performance in expression accuracy, detail preservation, and identity consistency, with efficient training and real-time rendering, making it suitable for AR/VR, gaming, and multimedia.
Creating highly realistic and animatable 3D head avatars is a critical area in computer graphics and 3D vision, with significant applications in augmented and virtual reality (AR/VR), gaming, and multimedia content creation. These digital representations need to accurately capture head geometry, facial dynamics, and offer fine-grained control with real-time rendering capabilities.
Recent advancements in 3D Gaussian Splatting (3DGS) have shown great promise in modeling complex 3D scenes and enabling real-time rendering. However, existing 3DGS-based methods for head avatar reconstruction still face challenges in capturing subtle facial expressions and maintaining continuous local textures, especially in areas that deform significantly, like the mouth and eyebrows.
Introducing EAvatar: A New Approach to Head Avatar Reconstruction
To overcome these limitations, researchers have proposed a novel 3DGS-based framework called EAvatar. This new method is designed to be both expression-aware and deformation-aware, leading to more accurate and visually coherent head reconstructions with improved expression control and detail fidelity.
EAvatar introduces several key innovations:
-
Sparse Expression Control: The method uses a small number of ‘key Gaussians’ to influence the deformation of their neighboring Gaussians. This allows for precise modeling of local deformations and smooth texture transitions, which is crucial for realistic facial expressions.
-
Generative Geometry Priors: EAvatar leverages high-quality 3D structural information from pre-trained generative models. This provides reliable facial geometry, offering structural guidance that improves the stability and accuracy of the avatar’s shape during the training process.
Traditional methods, such as mesh-based 3D Morphable Models (3DMMs), often struggle with fine-grained local edits because they rely on global linear blend weights for expression control. Neural implicit function-based approaches like NeRF, while capable of continuous fields, can have difficulty capturing high-frequency geometric details and maintaining local coherence in highly deformable regions, leading to issues like texture blurring.
Existing 3DGS-based methods for head avatars, while offering high rendering efficiency, also have their drawbacks. Some lack fine-grained control over local areas, while others struggle with generalizing to extreme expressions or are limited by the low-frequency motion of underlying mesh structures.
How EAvatar Works
EAvatar addresses these challenges by integrating both expression-aware and deformation-aware control mechanisms. It identifies key Gaussians that undergo substantial expression-induced deformation and then uses a spatial propagation strategy to adjust their neighbors. This ensures more precise and localized control in expressive regions.
Furthermore, EAvatar includes a ‘Gaussian splitting strategy’. When a Gaussian’s predicted displacement exceeds a certain threshold, it is dynamically split into two instances. This increases the local density of Gaussians in highly deformable areas, allowing for a finer representation of complex geometric variations while maintaining computational efficiency.
To ensure a stable and accurate foundation, EAvatar incorporates a structure-aware geometry modeling stage. This stage uses a high-quality prior mesh generated by a large-scale pre-trained generative model. This prior guides the optimization of an implicit signed distance function (SDF), from which a differentiable mesh surface is extracted. This process provides reliable and identity-aware guidance for initializing the Gaussians in the subsequent stages.
The training process for EAvatar is divided into two stages: first, structural geometry modeling, which focuses on stable geometry initialization guided by the generative prior; and second, dynamic optimization, which refines the appearance and expression control of the avatar.
Also Read:
- RealTalk: Crafting Realistic Emotional Digital Humans
- GeoSAM2: Precise 3D Object Segmentation with Simple 2D Prompts
Performance and Efficiency
Experimental results demonstrate that EAvatar produces more accurate and visually consistent head reconstructions. It shows superior performance in expression reconstruction accuracy, detail preservation, and identity consistency across various benchmarks, including self-reenactment and cross-identity reenactment tasks. The method also generalizes well to novel viewpoints, maintaining 3D consistency.
In terms of efficiency, EAvatar’s training time is approximately 2.5 days per subject on a single NVIDIA RTX 3090 GPU, which is faster than some comparable state-of-the-art methods. At inference time, EAvatar achieves real-time performance of approximately 32 frames per second (FPS), making it suitable for interactive and near real-time applications.
This innovative framework represents a significant step forward in creating high-fidelity, expression-driven 3D head avatars, offering enhanced realism and control for a wide range of digital applications. For more technical details, you can refer to the full research paper here.
