TLDR: STANCE is a novel image-to-video framework designed to generate videos with physically plausible object motion and interactions. It addresses common challenges in video generation by introducing ‘Instance Cues’ to convert sparse user inputs (like 2D arrows, mass, and depth delta) into dense 2.5D motion fields, and ‘Dense RoPE’ to maintain the precision and effectiveness of these cues during processing. By jointly predicting RGB video and structural maps (segmentation or depth), STANCE stabilizes optimization, improves temporal coherence, and allows users to control object trajectories, speeds, and collision outcomes with high fidelity, outperforming previous methods in motion coherence.
Creating videos where objects move and interact realistically has been a significant challenge in the field of artificial intelligence. While AI models have made impressive strides in generating visually stunning videos, maintaining consistent object trajectories, realistic inertia, and believable interactions often remains elusive. This is particularly true when the guidance comes from simple, human-provided motion hints, which can sometimes be too sparse to effectively steer the complex dynamics of a video.
Researchers have identified two key bottlenecks: first, simple 2D motion hints often lose their effectiveness after being processed by the AI, becoming too vague to guide the video generation accurately. Second, models that try to optimize for both visual appearance and motion consistency at the same time can sometimes prioritize looking good over moving correctly, leading to unnatural or inconsistent movements.
Introducing STANCE: A New Approach to Coherent Video Generation
A new framework called STANCE (MOTIONCOHERENTVIDEO GENERATIONVIA SPARSE-TO-DENSEANCHOREDENCODING) aims to overcome these challenges. STANCE is an image-to-video framework designed to produce videos with striking visual quality while ensuring coherent object motion and interactions. It achieves this through two main components: Instance Cues and Dense RoPE, paired with a unique joint prediction strategy.
Instance Cues: Making User Input More Powerful
STANCE introduces ‘Instance Cues’ as a pixel-aligned control signal. Imagine you want to tell an AI to move a specific object in a video. Instead of just drawing a simple 2D arrow, STANCE allows users to provide a keyframe (a single image), draw coarse 2D arrows to indicate direction and speed, assign a mass to each object, and even specify a scalar depth delta (∆z) to disambiguate motion in or out of the camera’s view. These sparse inputs are then transformed into a dense, 2.5D motion field that is camera-relative, meaning it understands motion in three dimensions relative to the camera. This makes the guidance much more precise and reduces ambiguity, especially when the camera itself is moving.
Dense RoPE: Preserving Motion Detail
The second innovation is ‘Dense RoPE’ (Rotary Position Embedding). A common problem is that when motion hints are processed by the AI, especially for small or thin objects, they can become diluted or lost. Dense RoPE addresses this by selecting the most important spatial locations within the user-defined motion regions and tagging a fixed budget of motion tokens with special spatial-addressable rotary embeddings. These tags act like stable, high-signal anchors that help the AI model maintain the spatial identity and precise alignment of the motion cues over time, even for tiny objects. This ensures that the control signals remain effective and token-dense throughout the generation process.
Joint RGB + Auxiliary Map Generation: Stabilizing Motion and Appearance
STANCE also employs a strategy of jointly generating the RGB (color) video alongside an auxiliary structural map, such as segmentation or depth. Both streams share spatio-temporal tokens and attend to the same motion cues. The structural head acts as a ‘geometry/consistency witness,’ helping to regularize the RGB head. This means the structural information anchors the geometry and consistency of the scene, while the RGB head focuses on the visual appearance. This joint prediction stabilizes the optimization process, leading to improved temporal coherence and reduced drift without needing detailed, frame-by-frame trajectory scripts.
Also Read:
- SceneAdapt: Integrating Scene Understanding into Motion Generation
- CanvasMAR: A Novel Approach for Enhanced Video Generation with Global Frame Prediction
Real-World Applications and Performance
The framework allows for physically meaningful edits while preserving the appearance of objects. For example, users can increase an object’s mass to reverse collision outcomes, specify larger speeds for longer travel distances, or rotate an arrow to reorient trajectories and shift contact points. These controls yield predictable kinematics and coherent chain reactions in multi-object scenes. STANCE has been trained on a comprehensive dataset of 200,000 clips featuring rigid-body interactions in both simple and realistic environments.
Evaluations using a metric called ‘Physics IQ,’ which specifically assesses motion coherence and interaction plausibility, show that STANCE significantly outperforms existing video generation baselines. It produces cleaner contact onsets, fewer ‘hovering’ frames before or after impact, and maintains object identity across frames. The effectiveness of Instance Cues and Dense RoPE, as well as the benefits of joint auxiliary generation, have been confirmed through extensive studies.
STANCE has also been successfully demonstrated in real-world scenarios, such as simple tabletop collisions captured with a smartphone. It accurately follows user-specified directions and speeds, preserves object identity, and generates physically coherent outcomes, including domino-like chain reactions with plausible timing and momentum transfer.
While STANCE represents a significant leap forward, it does have some limitations. Its training set currently doesn’t include fixed boundaries like walls or table edges, meaning interactions requiring wall contact and elastic rebound are not fully supported. It also focuses on rigid objects, so highly non-rigid objects like cloth or liquids are outside its current scope. Despite these, STANCE offers a powerful and intuitive way to generate videos with unprecedented control over object motion and interactions. For more technical details, you can refer to the full research paper. Read the full research paper here.
