TLDR: A new framework called “Virtually Being” introduces a method for highly realistic and customizable video generation, specifically designed for virtual production. It leverages multi-view performance captures and 4D Gaussian Splatting to create rich training data, enabling video diffusion models to achieve consistent character identity across various camera angles, precise 3D camera control, and adaptable lighting. The system also supports multi-subject generation, scene customization, and control over motion and spatial layout, offering a versatile tool for filmmakers to create high-fidelity virtual content.
The world of filmmaking is undergoing a significant transformation with the rise of virtual production, a process that blends live-action footage with computer-generated elements. This innovative approach offers filmmakers unprecedented control over scene layout, lighting, character performance, and camera movement. At the forefront of this evolution are advanced video generation models, which are increasingly being integrated into the pipeline to unlock new creative possibilities and streamline content creation.
A key challenge in this domain is the ability to customize videos with specific subjects while maintaining their identity from multiple viewpoints, especially when the camera is in motion. Traditional methods often fall short, relying on single reference images that fail to capture a subject’s full multi-view characteristics, leading to inconsistencies when viewed from different angles. Furthermore, existing camera control techniques are often limited, offering only basic 2D movements rather than the complex 3D camera trajectories common in professional filmmaking.
Addressing these limitations, a team of researchers from Eyeline Labs and Netflix has introduced a groundbreaking framework called “Virtually Being: Customizing Camera-Controllable Video Diffusion Models with Multi-View Performance Captures.” This novel approach enables both multi-view character consistency and precise 3D camera control within video diffusion models through an innovative data pipeline.
The core of their method lies in repurposing 4D Gaussian Splatting (4DGS) from volumetric captures as a data generator. Volumetric capture involves recording dynamic human performances using sophisticated multi-camera rigs, such as a 75-camera facial setup or a 160-camera full-body system. From these captures, 4DGS reconstructs dynamic human motion and renders videos along diverse, precisely annotated camera trajectories. To further enhance realism, the data is augmented with lighting variations using a generalizable video relighting model, producing a rich dataset with multi-view character supervision, accurate camera conditioning, and lighting diversity.
The framework employs a two-stage training strategy. First, a base model is pretrained on general camera-annotated datasets to learn camera-conditioned video generation. Then, this model is fine-tuned on the subject-specific multi-view data generated by 4DGS. This process ensures strong multi-view identity preservation, precise camera control, and lighting adaptability, allowing the model to generate identity-consistent, camera-controllable videos of customized subjects in diverse and unseen contexts.
Beyond single-subject generation, “Virtually Being” supports a range of features essential for virtual production. It enables multi-subject generation through two approaches: joint training on combined datasets or a more efficient noise blending technique that combines independently customized models at inference time. The framework also facilitates scene customization using high-quality computer-generated videos and even real-life videos captured with iPhones, where camera poses are estimated. Additionally, it offers control over subject motion and spatial layout by fine-tuning models like Go-with-the-Flow, allowing users to preserve movement patterns from source videos while replacing subjects with customized ones.
Also Read:
- Achieving Consistent Multi-View Customization with MVCustom
- SceneAdapt: Integrating Scene Understanding into Motion Generation
Extensive experiments and user studies have demonstrated significant improvements in video quality, personalization accuracy, camera control, and lighting adaptability compared to existing methods. The framework’s ability to maintain consistent subject identity across varying viewpoints and accurately follow complex 3D camera paths marks a substantial leap forward in integrating video generation into virtual production workflows. While the current implementation has limitations, such as the need for fine-tuning and the resolution of the underlying video diffusion model, “Virtually Being” offers a scalable and flexible solution for creating controllable, high-fidelity video content. You can read the full research paper here.
