TLDR: Kaleido is an open-source subject-to-video (S2V) generation framework that addresses key challenges in creating consistent videos from multiple reference images. It introduces a comprehensive data construction pipeline for diverse, high-quality training data and a novel Reference Rotary Positional Encoding (R-RoPE) to precisely integrate multi-image conditions. Kaleido significantly outperforms existing models in subject consistency, background disentanglement, and overall video quality, making it a state-of-the-art solution for generating subject-consistent videos.
In the rapidly evolving field of artificial intelligence, video generation models are making significant strides, promising to transform content creation. Among these, Subject-to-Video (S2V) generation stands out, aiming to create dynamic, consistent videos based on specific subjects provided through reference images. This capability holds immense potential for industries like e-commerce and advertising, offering unprecedented control and flexibility.
However, existing S2V models, particularly those that are open-source, have faced considerable challenges. They often struggle to maintain visual consistency across multiple subjects within a video and to effectively separate the subject from its background. This can lead to issues like subjects appearing inconsistently or unwanted background elements from reference images bleeding into the generated video. These limitations stem from two primary areas: a scarcity of diverse and high-quality training data, and less-than-optimal strategies for integrating multiple reference images.
Introducing Kaleido: A New Framework for Subject-to-Video Generation
A team of researchers from Hefei University of Technology, Tsinghua University, and Zhipu AI, including Zhenxing Zhang, Jiayan Teng, Zhuoyi Yang, Tiankun Cao, Cheng Wang, Xiaotao Gu, Jie Tang, Dan Guo, and Meng Wang, has introduced Kaleido. This innovative, open-source framework is designed to overcome the persistent challenges in multi-subject S2V generation, aiming to synthesize videos that are highly consistent with target subjects provided through multiple reference images.
Addressing the Data Challenge
One of Kaleido’s core innovations is its dedicated data construction pipeline. Recognizing that the quality and diversity of training data are crucial, the researchers developed a comprehensive approach that includes:
- Multi-Class Sampling and Filtering: This ensures a rich variety of subjects and scenes, while stringent filters remove low-quality samples.
- Cross-Paired Data Construction: This technique helps the model learn to disentangle subjects from their backgrounds, preventing extraneous elements from being carried over into the generated videos.
- Background Disentanglement: Advanced inpainting techniques are used to effectively erase background information from reference images, encouraging the model to reconstruct subject appearances based on the images and synthesize backgrounds from textual prompts.
- Pose and Motion Enrichment: Utilizing tools like Flux Redux, the pipeline enriches reference images with new poses and motions, helping the model learn a more generalizable representation of subject identity.
Enhancing Multi-Image Integration with R-RoPE
To tackle the issue of inadequate conditioning strategies, Kaleido introduces Reference Rotary Positional Encoding (R-RoPE). When multiple reference images are provided, the model needs to understand that these are distinct inputs, not sequential video frames. R-RoPE achieves this by assigning unique positional vectors to image tokens, spatially shifting them to occupy distinct positions from video tokens. This explicit separation prevents the model from misinterpreting image conditions as part of the video sequence, thereby improving multi-image and multi-subject consistency without adding significant computational cost.
Also Read:
- HumanCM: Accelerating Human Motion Prediction with Single-Step Generation
- MUG-V 10B: Advancing High-Efficiency Training for Large Video Generation
Performance and Impact
Extensive experiments demonstrate that Kaleido significantly surpasses previous methods in terms of consistency, fidelity, and generalization. It achieves top scores in critical metrics like S2V Consistency, which measures how well subject identity is preserved, and S2V Decoupling, which evaluates the model’s ability to disentangle background information. User studies further confirm Kaleido’s superiority, with human raters consistently preferring its output in terms of video quality, prompt alignment, and subject consistency.
By open-sourcing both its data pipeline and pretrained S2V model, Kaleido provides a robust foundation for future research in subject-to-video generation, pushing the boundaries of what’s possible in AI-driven video creation. You can find more details about this research in the full paper: KALEIDO: OPEN-SOURCEDMULTI-SUBJECTREFER-ENCEVIDEOGENERATIONMODEL.
