TLDR: A new framework called “Pack and Force Your Memory” introduces MemoryPack for dynamic context modeling to achieve minute-level temporal consistency and Direct Forcing to mitigate error accumulation by aligning training and inference. This approach significantly enhances the reliability and consistency of long-form video generation, achieving state-of-the-art performance.
Generating long-form videos, especially those lasting minutes, has been a significant challenge in the field of generative AI. Current models often struggle with two main issues: maintaining consistency over long durations and preventing errors from accumulating as the video progresses. Imagine trying to generate a minute-long video of a person walking – ensuring the person’s appearance, the background, and the motion remain consistent throughout is incredibly complex.
Researchers from ShanghaiTech University, Tencent Hunyuan, and Nanjing University have introduced a new framework called “Pack and Force Your Memory” to tackle these challenges. Their work focuses on creating more consistent and reliable long-form video generation, making autoregressive video models more practical for real-world applications. You can find the full research paper here: Pack and Force Your Memory: Long-form and Consistent Video Generation.
Addressing Long-Range Consistency with MemoryPack
One of the core innovations is MemoryPack, a clever mechanism designed to handle dynamic context modeling. When generating a long video, it’s crucial for the model to remember what happened much earlier in the sequence, not just the most recent frames. MemoryPack acts like a sophisticated memory system, using both textual descriptions and image information as global guides. This allows it to understand and model both short-term details (like immediate motion) and long-term dependencies (like object identity or scene layout) over minute-long periods.
Unlike older methods that might just keep a small window of recent frames or aggressively compress information, MemoryPack offers a flexible way to associate historical information with future frame generation. It does this efficiently, ensuring that the computational cost doesn’t explode as video length increases. This means it can maintain temporal consistency without becoming too slow or resource-intensive.
Mitigating Error Accumulation with Direct Forcing
The second major hurdle in long-form video generation is error accumulation. During training, models learn by looking at perfect, “ground-truth” frames. However, during actual video generation (inference), they have to rely on their own predictions from previous steps. This mismatch can cause small errors to compound over time, leading to noticeable “drift” where objects change appearance, backgrounds shift, or motion becomes unnatural.
To combat this, the researchers introduced Direct Forcing. This is an efficient, single-step strategy that helps align the training process with how the model will actually generate videos during inference. By approximating inference outputs during training, Direct Forcing exposes the model to more realistic conditions. This significantly reduces the discrepancy between training and inference, thereby curtailing the propagation of errors and leading to much more stable and consistent long-form videos.
Also Read:
- Achieving Minute-Scale High-Quality Video Generation with Self-Forcing++
- Enhancing Video Generation with Precise Temporal Control
Achieving State-of-the-Art Performance
The combination of MemoryPack and Direct Forcing has yielded impressive results. The new method has achieved state-of-the-art performance on various benchmarks, particularly excelling in metrics related to motion smoothness, background consistency, and subject consistency. This means the generated videos are not only visually appealing but also maintain a high degree of coherence and stability over extended durations. Human evaluations also ranked this approach as superior, further validating its effectiveness in producing high-quality, consistent long-form videos.
This research marks a significant step forward in making long-form video generation more practical and reliable, opening doors for advanced applications in content creation, embodied intelligence, and interactive gaming.
