TLDR: MaskCaptioner is a new AI model that can simultaneously detect, segment, track, and describe objects in videos using natural language. It achieves this by training on newly generated synthetic datasets (LVISCap and LV-VISCap) created with a Vision Language Model, leading to state-of-the-art performance on key benchmarks and extending video object captioning to include segmentation.
Understanding videos with human-like precision is a core goal in computer vision. This involves not just seeing objects, but also knowing where they are, how they move, and what they are doing, then describing all of this in natural language. This complex task is known as Dense Video Object Captioning (DVOC).
Traditionally, DVOC has faced significant hurdles, primarily due to the immense cost and effort required for manual annotation. Imagine having to meticulously label every object, its movement, and describe its actions in every frame of a video! This has led previous methods to use fragmented training strategies, often resulting in less-than-optimal performance.
A new research paper introduces MaskCaptioner, an innovative end-to-end model designed to overcome these challenges. MaskCaptioner learns to jointly detect, segment, track, and caption object trajectories in videos. This means it can identify an object, outline its exact shape, follow it through the video, and then generate a descriptive sentence about its actions and appearance.
The key to MaskCaptioner’s success lies in its novel approach to data generation. The researchers leveraged a powerful Vision Language Model (VLM), specifically Gemini 2.0 Flash, to create synthetic, object-level captions. They extended existing segmentation datasets, LVIS (for images) and LV-VIS (for videos), with these AI-generated captions, creating two new datasets: LVISCap and LV-VISCap. These datasets are unique because they provide comprehensive (mask, box, category, caption) annotations for all objects, enabling a unified training approach.
The MaskCaptioner architecture is built upon a state-of-the-art Open-Vocabulary Video Instance Segmentation (OV-VIS) model, OVFormer, and extends it with a specialized captioning head. It processes videos in clips, identifying and segmenting objects, then uses a sophisticated tracking module to follow these objects across the entire video. Finally, a captioning head, based on BLIP-2, generates a single, coherent caption for each tracked object trajectory, describing its actions and appearance throughout the video.
Training MaskCaptioner on these synthetically generated datasets has yielded impressive results. The model significantly outperforms previous state-of-the-art methods on three major DVOC benchmarks: VidSTG, VLN, and BenSMOT. Notably, it not only improves detection and tracking but also achieves substantial gains in captioning accuracy. Furthermore, MaskCaptioner extends the DVOC task to include segmentation masks, providing a more granular understanding of objects in videos.
The research also highlights the importance of the generated data scale, showing that more training captions lead to better captioning performance. The temporal aggregation module, which merges information from multiple video clips, further enhances the richness and accuracy of the generated captions, especially for longer videos with complex actions.
Also Read:
- Achieving Consistent Multi-View Customization with MVCustom
- VLA-0: Achieving Advanced Robot Control with Unmodified Vision-Language Models
This work represents a significant step forward in video understanding, offering a unified and efficient way to analyze and describe dynamic scenes. The datasets and code for MaskCaptioner are publicly available, paving the way for future advancements in this exciting field. You can read the full research paper here.
