TLDR: SceneAdapt is a new framework that enhances text-conditioned human motion generation models by making them aware of their surrounding 3D scenes. It achieves this through a two-stage adaptation process: first, by teaching the model motion inbetweening with a novel Context-aware Keyframing (CaKey) layer, and then by adding scene-conditioning layers that learn to interact with scene geometry using patch-wise features. This approach allows for the creation of diverse, semantically rich, and physically consistent human motions without needing large, expensive datasets that combine text, motion, and scene information.
Generating realistic human motion for virtual reality, gaming, and robotics has long been a complex challenge. Current approaches often fall short in one of two critical areas: either they produce motions that are semantically rich and diverse (meaning they understand what the action is, like ‘walking’ or ‘jumping’) but are oblivious to the surrounding environment, leading to characters walking through walls; or they generate scene-consistent motions but are limited to a narrow range of actions due to the difficulty of collecting large, diverse datasets with precise scene interactions.
A new framework called SceneAdapt, developed by researchers from Yonsei University, Carnegie Mellon University, UC Merced, Google DeepMind, and Seoul National University, aims to bridge this gap. SceneAdapt injects scene awareness into existing text-conditioned motion models, allowing them to generate motions that are both semantically meaningful and physically consistent with their environment. Crucially, it achieves this without the need for massive, expensive datasets that combine text, motion, and scene information, which are notoriously difficult to collect.
How SceneAdapt Works: A Two-Stage Adaptation
The core idea behind SceneAdapt is to use ‘motion inbetweening’ as a proxy task. Motion inbetweening involves generating the intermediate frames of a motion sequence given a few keyframes (like start and end poses). This task can be learned without text, making it a perfect bridge between different types of datasets.
The framework operates in two distinct stages:
1. Stage 1: Motion Inbetweening Adaptation. In this initial stage, SceneAdapt takes a pre-trained text-to-motion model and adapts it for motion inbetweening. It introduces novel ‘Context-aware Keyframing (CaKey) layers’. These layers selectively modulate the motion data based on the provided keyframes, ensuring that the generated motion accurately connects the key poses while preserving the naturalness and semantic richness of the original model. The CaKey layers are designed to be ‘context-aware’, meaning they consider the keyframe signal, the diffusion timestep, and the current motion representation to make precise adjustments. They also employ ‘sparse modulation’, applying changes only to the keyframe indices to avoid distorting non-keyframe parts of the motion.
2. Stage 2: Scene-Aware Inbetweening. Building on the inbetweening capability, SceneAdapt then adds ‘scene-conditioning layers’. While the CaKey layers from Stage 1 are frozen, these new layers are trained using scene-motion pairs. Instead of relying on a single, global representation of the scene, SceneAdapt uses ‘patch embeddings’ from a voxel ViT (Vision Transformer). This allows the model to dynamically attend to specific local areas of the scene that are relevant to the human’s movement at any given moment, capturing spatio-temporal variations more effectively. This stage focuses exclusively on leveraging scene information to achieve scene-consistent inbetweening, thereby injecting scene awareness into the model.
Scene-Aware Text-to-Motion Generation
Once both adaptation stages are complete, SceneAdapt can generate motions conditioned on both text prompts and 3D scene inputs. During inference, the model uses an all-zero keyframe mask, meaning no keyframes are provided, and it relies solely on the text and scene information. The framework also incorporates ‘classifier-free guidance’ scales, allowing users to control the trade-off between how closely the motion aligns with the text description and how physically consistent it is with the scene.
Also Read:
- MimicParts: Advancing Realistic 3D Motion Generation from Speech
- Mesh-Gait: Advancing Gait Recognition with Efficient 3D Reconstruction from 2D Silhouettes
Key Advantages and Results
Experiments demonstrate that SceneAdapt significantly improves scene awareness in text-to-motion models without sacrificing their ability to generate diverse and semantically rich actions. Compared to existing methods, SceneAdapt achieves superior performance in both text-to-motion alignment and scene consistency. It also offers a practical advantage: it is orders of magnitude faster than optimization-based approaches, which can be very slow. The use of voxel patch features for scene representation, as opposed to global class embeddings, proved more effective, allowing motion latents to interact with scene patches in a spatially adaptive manner.
In essence, SceneAdapt provides a robust and efficient solution for creating digital human motions that are not only expressive and natural but also intelligently interact with their surrounding environments, paving the way for more immersive virtual experiences. You can find more details about this research in the full paper: SCENEADAPT: SCENE-AWARE ADAPTATION OF HUMAN MOTION DIFFUSION.
