TLDR: VisualSpeaker is a novel method for creating highly realistic 3D avatar lip animations from speech or text. It introduces a unique ‘perceptual lip-reading loss’ that directly evaluates the visual quality of rendered lip movements using photorealistic 3D Gaussian Splatting and a Visual Automatic Speech Recognition model. This approach significantly improves lip synchronization and clarity, leading to more expressive and intelligible avatars, with particular benefits for applications like sign language production where precise mouthings are essential.
Creating realistic and expressive 3D facial animations for digital avatars is a complex challenge, especially as these avatars are increasingly used in areas like telepresence, interactive media, and crucial applications such as Sign Language Production (SLP). People are naturally adept at noticing subtle facial cues, making it difficult to animate photorealistic 3D head avatars convincingly.
Traditional methods for 3D talking heads often rely on geometric accuracy, focusing on making predicted mesh vertices closely match ground truth. While this reduces geometric error, it can lead to animations that appear averaged and lack the subtle expressiveness needed to differentiate visemes – the basic visual units of speech that represent distinct mouth shapes and movements. This is particularly problematic for sign language, where mouthings are primary linguistic components used to distinguish similar manual signs.
Introducing VisualSpeaker
A new method called VisualSpeaker aims to bridge this gap between geometric accuracy and perceptual intelligibility. It introduces a novel approach that directly optimizes for lip readability in the rendered pixel space. Unlike prior work that computes losses on intermediate mesh representations, VisualSpeaker leverages photorealistic differentiable rendering, specifically 3D Gaussian Splatting (3DGS), to supervise the final visual output.
The core innovation is a ‘perceptual lip-reading loss’ derived by feeding photorealistic 3D Gaussian Splatting avatar renders through a pre-trained Visual Automatic Speech Recognition (V-ASR) model during training. This allows the system to receive feedback on how well the generated lip movements are perceived, much like a human would interpret them.
VisualSpeaker uses an encoder-decoder transformer architecture, inspired by Faceformer, to generate 3D facial animations from either audio or text input. It employs the FLAME parametric 3D face model for consistent head geometry and integrates 3DGS for high-fidelity, real-time rendering. The training process involves a three-stage curriculum, starting with geometric pretraining, then adapting to more complex datasets, and finally introducing the perceptual lip-reading loss for fine-tuning.
Also Read:
- Interactive Sound Generation: Click on Objects, Hear the Audio
- Unveiling the Physical Basis of Accents: A Look at Tongue and Lip Movements
Key Improvements and Applications
Evaluations on the MEAD dataset demonstrate that VisualSpeaker significantly improves both standard Lip Vertex Error (LVE) metrics and the perceptual quality of generated animations. The LVE metric, which measures the geometric accuracy of lip movements, saw a 56.1% improvement. While common visual quality metrics like PSNR and SSIM showed modest gains, a user study confirmed that participants overwhelmingly preferred VisualSpeaker’s animations for their realism and lip clarity, especially when compared to models trained without the perceptual lip-reading loss.
A significant application of VisualSpeaker is in Sign Language Production. The method can generate linguistically meaningful mouthings for signs that share the same manual component but differ only in their mouthings. For example, it can produce distinct mouth shapes for signs like ‘why’ and ‘because’ in British Sign Language (BSL) purely from text input and synthesized speech, without needing paired audio or manual alignment. This capability is crucial for accessibility systems and human-computer interaction, enabling more accurate and intelligible sign language avatars.
While VisualSpeaker marks a promising step forward, the researchers acknowledge limitations such as the high computational cost of differentiable 3DGS training and the reliance on per-subject multi-view avatars. Future work could explore more efficient renderers, generalizable avatar pipelines, and explicit control over emotion and upper-face cues to achieve even greater human-level realism.
For more technical details, you can read the full research paper here.
