TLDR: The JWB-DH-V1 benchmark introduces a large dataset and evaluation protocols to address challenges in generating realistic whole-body talking avatars with synchronized speech. It provides comprehensive metrics for visual and audio quality, revealing performance gaps in current models for full-body animation compared to face/hand-centric approaches, guiding future research in this field.
Creating digital humans that can move and speak naturally, especially with full-body motion synchronized with speech, has been a significant challenge in the field of artificial intelligence. While recent advancements in AI-driven video generation have made impressive strides in creating short, realistic video clips, they often fall short when it comes to achieving consistent quality across both visual and audio elements, particularly for whole-body animations.
Current methods tend to focus on specific areas like faces or upper bodies, neglecting the complexities of full-body movements and gestures that naturally accompany speech. Furthermore, there’s a noticeable absence of comprehensive evaluation frameworks that can assess both the visual and audio quality of these generated avatars, and a lack of benchmarks for analyzing performance in different body regions.
Introducing JWB-DH-V1: A New Standard for Digital Human Generation
To bridge these critical gaps, researchers have introduced the Joint Whole-Body Talking Avatar and Speech Generation Version I (JWB-DH-V1). This groundbreaking benchmark aims to push the boundaries of what’s possible in creating realistic and synchronized digital humans. JWB-DH-V1 is not just a dataset; it’s a complete ecosystem designed to evaluate and advance the state-of-the-art in joint audio-video generation for whole-body animatable avatars.
At its core, JWB-DH-V1 features a massive multi-modal dataset. It comprises an impressive 10,000 unique identities, captured across 2 million video samples. This vast collection provides an unprecedented resource for training and testing AI models. Each sample in the dataset is richly annotated, including details like body segmentation, landmarks, bounding boxes for key regions (hands, legs, whole body), motion descriptions, speech transcriptions, and ground truth audio with word boundaries. This meticulous annotation ensures a fine-grained understanding of the alignment between body postures, hand gestures, leg stance, and spoken language over time.
Beyond the dataset, JWB-DH-V1 establishes a robust evaluation protocol. This protocol is designed to comprehensively assess the quality of generated whole-body avatars and their synchronized speech. It breaks down the evaluation into several key areas:
- Video Generation Evaluation: This part uses six reference-free metrics to gauge the perceptual quality and temporal stability of the generated video. These include Subject Consistency (how consistent the avatar’s appearance is over time), Background Consistency (how stable the background remains), Motion Smoothness (how fluid the movements are), Dynamic Degree (how rich and varied the motion is), Aesthetic Quality (how visually pleasing the output is), and Imaging Quality (the clarity of the visuals).
- Co-Speech Evaluation: To ensure the generated avatars are realistic and coherent, six standard metrics are used to assess frame quality, temporal coherence, and identity preservation. These metrics help determine how well the avatar’s movements align with the speech and if the avatar’s identity is maintained throughout the video.
- Speech Audio Evaluation: The quality of the generated speech audio is crucial. JWB-DH-V1 employs advanced Large-Audio-Language Models (LALMs) for evaluation, going beyond traditional metrics like Word Error Rate (WER). This allows for a more nuanced assessment of aspects like prosody (the rhythm and intonation of speech), pausing, expressiveness, and pronunciation, using a “win-rate” based metric to compare candidate systems against strong baselines.
Also Read:
- LOTUS: A New Framework for Evaluating Advanced Image Captioning
- Enhancing Spoken Language Correction with Automated Data Generation
Initial Findings and Future Directions
The initial evaluation of several state-of-the-art models using JWB-DH-V1 has already yielded significant insights. The results consistently show performance disparities, indicating that models perform better when generating face or hand-centric animations compared to full whole-body performance. This highlights crucial areas where future research needs to focus to achieve truly integrated and realistic full-body talking avatars.
The introduction of JWB-DH-V1 marks a pivotal moment in the development of digital humans. By providing a large-scale public dataset and comprehensive evaluation protocols, it sets a new standard for research in synchronized and high-quality audio-video generation. Future iterations of the benchmark are planned to support longer video clips (up to 60 seconds) and further refine audio-video annotations, continuing to push the boundaries of this exciting field. The dataset and evaluation tools are publicly available for researchers to explore and contribute to at https://github.com/deepreasonings/WholeBodyBenchmark.
