TLDR: A study compared the evaluation of AI-generated gestures in 2D versus Virtual Reality (VR) settings. It found that gestures observed in VR were rated slightly higher on average, suggesting that VR enhances the perception of human-likeness and appropriateness. While VR offers benefits like increased participant preference and potentially a more robust ground-truth comparison, it also presents drawbacks such as higher hardware requirements and time consumption. The research consistently ranked three gesture generation models, with motion-captured data (GT) performing best, followed by SG, SF, and SJ, indicating that model quality ranking remains stable across settings.
A recent study delves into the evolving landscape of evaluating AI-generated gestures, moving beyond traditional 2D screens to explore the immersive potential of Virtual Reality (VR). The paper, titled “Gesture Evaluation in Virtual Reality,” by Axel Wiebe Werner, Jonas Beskow, and Anna Deichler, investigates how the environment influences our perception of virtual gestures, offering crucial insights for the development of more lifelike and communicative AI avatars.
Traditionally, the assessment of gestures created by artificial intelligence has been confined to two-dimensional settings. However, with the rise of VR technology, researchers are now able to explore how a fully immersive 3D environment might change how we perceive these non-verbal cues. This research aimed to identify the differences, advantages, and disadvantages of evaluating gestures in VR compared to a standard 2D setup.
The study also put three gesture generation algorithms, submitted to the 2023 GENEA Challenge, to the test in both virtual settings. The goal was to see if their performance ranking remained consistent across environments and to understand the impact of immersion on their perceived quality.
Key Findings: VR’s Impact on Perception
The experiments revealed a significant finding: the VR setting does indeed influence how generated gestures are rated. Participants tended to rate gestures observed in VR slightly higher on average than those seen in 2D. This suggests that the immersive experience of VR might enhance the perception of gestures as more human-like and appropriate.
Interestingly, while the setting impacted the overall rating, the relative ranking of the gesture generation models remained consistent across both 2D and VR. This indicates that the inherent quality of the generation models was perceived similarly, regardless of the viewing environment. However, the VR setting had a more pronounced effect on the perception of ‘true movement,’ which received higher ratings in VR than in 2D.
Methodology: A Comparative Approach
To conduct this comparative evaluation, the researchers utilized data from the 2023 GENEA Challenge, which included machine-learned (ML) generated and motion-captured (ground truth) BVH data for avatar gesticulation, along with corresponding audio files. Three top-ranking systems from the GENEA 2023 human-likeness evaluation – SG (Diffusion-based), SF (Diffusion-based), and SJ (Transformer-based) – were selected for the study, alongside ground truth motion clips (GT).
The experiment involved 30 participants who observed avatars in three distinct scenarios, each evaluated once in 2D and once in VR:
- Scenario 1 (Naturalness): A single avatar with no audio, where participants rated the naturalness of its motion.
- Scenario 2 (Speech Appropriateness): A conversation between two avatars, with participants rating how appropriate the main agent’s gestures were to its speech.
- Scenario 3 (Dyadic Appropriateness): Similar to scenario 2, but participants focused on how appropriate the main agent’s gestures were to the interlocutor’s gestures and speech, assessing conversational flow.
Participants provided direct ratings after watching 12 clips for each scenario. The order of settings (2D vs. VR) was alternated between participants to minimize bias.
Discussion: Benefits and Drawbacks of VR Evaluation
The study’s discussion highlights that the slightly higher average rating in VR was statistically significant, confirming that VR affects the perception and rating of computer-generated gestures. This could be due to the enhanced social presence of avatars in VR, making them seem more human. Participants overwhelmingly preferred the VR setting, which could lead to increased engagement and focus during evaluations.
However, VR evaluations come with practical drawbacks, such as higher time consumption and hardware requirements. Tests had to be conducted in a lab setting, limiting scalability compared to online 2D evaluations. Despite these limitations, the fact that the ground truth (motion-captured) system consistently scored higher in VR suggests that VR might offer a more robust benchmark for comparing generated movements.
The performance of the generation models aligned with previous findings from the GENEA Challenge, with SG performing best, SF in the middle, and SJ performing least well among the AI models. The insignificant difference between GT and SG suggests that SG is performing very close to human-like motion capture.
Also Read:
- The Brain’s Surprising Embrace of AI Humor: Beyond Conscious Bias
- Vibe Coding: How AI is Reshaping UX Design Workflows and Collaboration
Future Directions
While the VR setting seemingly enhanced immersion and led to more positive perceptions, the researchers acknowledge the small sample size and demographic similarities as limitations. Future work should aim for larger, more diverse participant groups and could explore intermediate evaluation methods, such as 3D viewers without full immersion. Investigating participant engagement through measures like eye-tracking in different settings could also provide further valuable insights.
For more detailed information, you can access the full research paper here.
