TLDR: MIDAS is a novel framework for generating interactive digital human videos in real-time. Developed by the Kling Team at Kuaishou Technology, it utilizes an autoregressive large language model (LLM) to process multimodal inputs like audio, pose, and text, guiding a diffusion head to render high-quality video frames. The system features a deep compression autoencoder for efficiency, a vast 20,000-hour dialogue dataset for training, and a noise injection strategy to mitigate errors. MIDAS enables low-latency, multimodal-controlled digital human synthesis, demonstrated through duplex conversations, multi-language video generation, and interactive world modeling, paving the way for more natural human-AI interactions.
The world of artificial intelligence is constantly pushing boundaries, and a recent development from the Kling Team at Kuaishou Technology, in collaboration with Zhejiang University and Tsinghua University, introduces a groundbreaking framework called MIDAS. This system aims to revolutionize how we interact with digital humans by enabling real-time, multimodal control over video generation.
Interactive digital human video generation has seen significant progress, but creating a practical system that can respond to various inputs in real-time has remained a challenge. Existing methods often struggle with delays, high computational costs, and limited control. MIDAS addresses these issues by offering an autoregressive video generation framework that supports interactive multimodal control and low-latency video extrapolation in a continuous stream.
What is MIDAS?
At its core, MIDAS leverages a standard large language model (LLM) with minimal modifications. This LLM is designed to accept a wide range of input signals, including audio, body pose, and text. It then processes these inputs to generate spatially and semantically coherent representations that guide a diffusion head, which is responsible for rendering the final high-quality video frames.
One of the key innovations in MIDAS is its ability to handle diverse input signals. Imagine a digital avatar that can not only speak based on text but also mimic your body movements and react to your voice in real-time. This level of multimodal control opens up exciting possibilities for applications such as virtual education, creative media, and more engaging human-AI interactions.
Overcoming Technical Hurdles
To achieve its impressive capabilities, MIDAS tackles several technical challenges:
- Low Latency and Efficiency: Traditional video generation methods, especially those based on diffusion transformers, often require extensive processing, making them unsuitable for real-time use. MIDAS uses an autoregressive approach, generating video frames sequentially based on past inputs, which significantly reduces computational overhead and enables faster generation.
- Deep Compression Autoencoder: To further enhance efficiency, the framework incorporates a deep compression autoencoder (DC-AE). This component can reduce the visual data by up to 64 times, effectively alleviating the computational burden on the autoregressive model, especially for long video sequences, while maintaining high reconstruction quality.
- Large-Scale Dialogue Dataset: Training such a sophisticated system requires vast amounts of data. The researchers constructed an extensive dialogue dataset of approximately 20,000 hours from various sources. This rich collection of conversational scenarios provides a strong foundation for training the model to generate natural and responsive digital human interactions.
- Mitigating Exposure Bias: A common problem in autoregressive models is “exposure bias,” where the model performs well on training data (ground truth) but struggles when generating its own sequences (which might have small errors that accumulate). MIDAS introduces a controlled noise injection mechanism during training, simulating imperfect predictions and teaching the model to recover from errors, thus improving long-term stability and quality.
How It Works: A Glimpse Under the Hood
The MIDAS architecture processes inputs in “chunks,” where each chunk contains condition tokens (audio, pose, text) followed by frame tokens. The autoregressive model predicts the next frame’s latent representation, which is then refined by a lightweight diffusion head. This diffusion head acts as a specialized renderer, transforming the model’s predictions into high-quality video frames with just a few denoising steps.
The system’s design allows for flexible-length, low-latency video generation, making it ideal for interactive applications. It can adapt to varying input qualities and maintain temporal consistency over long sequences, even up to an hour, with minimal degradation.
Also Read:
- EAI-Avatar: Crafting Emotionally Responsive Virtual Conversations
- VIBE VOICE: Advancing Long-Form, Multi-Speaker Speech Synthesis
Demonstrated Capabilities
The effectiveness of MIDAS has been validated through extensive experiments:
- Duplex Conversation: The system can enable natural, turn-taking dialogues between digital avatars. Each avatar exhibits appropriate listening behaviors and animates with synchronized lip movements and facial expressions when speaking.
- Multi-language Long Video Synthesis: MIDAS demonstrates precise phoneme-level synchronization across different languages, including Mandarin Chinese, Japanese, and English, without requiring language-specific training. It can generate videos up to 4 minutes long with consistent quality.
- General Interactive Video Generation: Beyond digital humans, the model’s architecture is flexible enough to accommodate arbitrary control signals. By reformulating multimodal conditions into directional controls and training on datasets like Minecraft, MIDAS can function as a real-time interactive world model, showing strong visual consistency and memory capabilities.
The MIDAS framework represents a significant step forward in the field of interactive digital human synthesis. By combining an autoregressive LLM with efficient compression and a diffusion head, it offers a powerful solution for creating responsive, high-quality, and multimodal-controlled digital avatars in real-time. This research paves the way for more immersive and natural human-AI interactions in the future. You can find more details about this innovative work in the full research paper available at this link.
