TLDR: MiVID is a novel, lightweight, self-supervised framework for video frame interpolation that uses a diffusion model. It combines a 3D U-Net with temporal attention and a hybrid masking strategy to learn robust spatiotemporal representations without needing high-frame-rate supervision or explicit motion estimation. Trained efficiently on CPU, MiVID achieves competitive and often superior results compared to supervised methods on datasets like UCF101-7 and DAVIS-7, demonstrating a scalable and accessible path for high-quality video enhancement.
Video Frame Interpolation (VFI) is a crucial technique in video enhancement, allowing for the creation of intermediate frames between existing ones. This process is vital for applications like slow-motion video, converting frame rates, and restoring older videos. Traditionally, VFI methods have relied on optical flow to estimate motion or on learning-based models that require extensive, perfectly labeled ground-truth data. However, these approaches often struggle with common real-world challenges such as occlusions (when objects block each other), sudden movements, and varying lighting conditions.
A new research paper introduces MiVID, a groundbreaking framework that tackles these limitations. MiVID stands out as a lightweight, self-supervised, and diffusion-based system for video interpolation. What makes MiVID particularly innovative is its ability to eliminate the need for explicit motion estimation, a complex and often error-prone step in many traditional VFI methods.
At its core, MiVID employs a compact 3D U-Net architecture, which is a type of neural network well-suited for processing volumetric data like video. This backbone is enhanced with transformer-style temporal attention, allowing the model to understand and focus on relevant information across different frames over time. This combination helps MiVID learn robust spatiotemporal representations – essentially, how objects move and change within a video scene.
A key innovation in MiVID is its hybrid masking regime. During training, the model is presented with video segments where certain frames are intentionally masked or obscured. This masking simulates real-world challenges like occlusions and motion uncertainty. MiVID uses three types of masking: random masking, where intermediate frames are randomly replaced with noise; motion-guided masking, which targets regions with high movement; and curriculum masking, where the amount of masking gradually increases as training progresses. By combining these strategies, MiVID learns to infer plausible motion and reconstruct missing frames without needing any high-frame-rate supervision, meaning it doesn’t require videos that were originally shot at very high frame rates for training.
Despite being a low-resource pipeline, trained entirely on CPU using datasets like UCF101-7 and DAVIS-7 with 9-frame video segments, MiVID achieves impressive results. It reaches optimal performance in just 50 epochs and is competitive with, and in some cases even surpasses, several supervised baseline models. This demonstrates the significant power of self-supervised diffusion priors for creating temporally coherent and realistic new frames.
Also Read:
- GEWDiff: A Novel Approach to Hyperspectral Image Super-resolution
- 4D3R: Reconstructing Dynamic Scenes from Single Videos Without Camera Pose Data
The framework’s ability to operate without dense supervision or explicit optical flow makes it highly adaptable to diverse motion patterns and scenarios with heavy occlusions. MiVID’s resource efficiency also makes it more accessible for broader research and practical applications, especially in settings where high-end computational infrastructure or extensive labeled data is limited. This work marks a significant step towards more accessible and generalizable VFI systems, showcasing how generative models can effectively enhance video quality. You can read the full research paper here.
