TLDR: Researchers have introduced MSC, a new large-scale dataset of marine wildlife videos. It features detailed annotations including segmentation masks and clip-level captions, designed to overcome challenges in marine video understanding. MSC enables improved AI models for tasks like video captioning, visual grounding, and text-to-video generation, providing a crucial resource for marine biology and environmental science.
Understanding the complex and dynamic world beneath the ocean’s surface poses significant challenges for artificial intelligence. Traditional video understanding datasets, often focused on general or human-centric scenarios, struggle to adapt to the unique complexities of marine environments, such as the unpredictable movements of marine objects, camera motion in water, and the intricate nature of underwater scenes. This limitation hinders the ability of AI to gain meaningful insights into marine life and ecosystems.
To address these critical gaps, a new research paper introduces a groundbreaking resource: MSC, a Marine Wildlife Video Dataset with Grounded Segmentation and Clip-Level Captioning. This innovative dataset aims to significantly advance marine video understanding and analysis, and even facilitate the generation of marine videos.
The core of MSC lies in its comprehensive video understanding benchmark, which uniquely combines three types of data: video footage, detailed text descriptions, and precise segmentation masks. This triplet approach enables visual grounding and captioning, providing a richer context for AI models. A key insight from the research is the effectiveness of splitting long videos into shorter, semantically coherent clips. This technique helps in detecting subtle object transitions and scene changes, thereby enriching the semantic content of the captions.
The MSC dataset is a substantial collection, comprising 24.8 hours of marine video content recorded from 13 different countries. It boasts fine-grained annotations, including clip-level textual descriptions meticulously provided by 18 biologists and pixel-level segmentation masks created by 20 professionals. This human expertise ensures high-quality and accurate data, crucial for training robust AI models.
The creation of MSC involved a meticulous two-stage annotation pipeline. First, annotators used a specialized web-based tool to segment marine objects, generating high-quality pixel-wise segmentation masks for six key categories: fish, reefs, aquatic plants, wrecks, human divers, and the sea floor. These manually refined masks were then used to identify target objects. The second stage focused on captioning. Recognizing that captions for long videos can be superficial, researchers split videos into short clips, each capturing a single-shot event. Large Language Models (LLMs) such as GPT-4.1, Gemini-2.0 Flash-Lite, and Qwen-VL were employed to generate initial textual descriptions for these clips. Crucially, these AI-generated descriptions were then refined by biologists to accurately reflect the semantic content and behaviors of segmented objects and their surrounding environment, mitigating the issue of AI hallucination.
The dataset also highlights challenges inherent in marine data, such as imbalances in object quantity and scale. For instance, while fish and coral reefs are numerous, fish are typically small objects, whereas reefs are large. Human divers are less frequent but consistently small, while wrecks are rare but large. These variations underscore the complexity of the marine domain.
The researchers conducted extensive benchmarking on MSC across various applications. For video-level captioning, models like Gemini-2.0 and MovieBench showed promising results. In clip-level captioning, GPT-4.1 demonstrated superior performance, with Gemini-2.0 also performing strongly. For visual grounding, which involves linking text queries to specific marine creatures or objects within videos, LLM-based models like VideoGLaMM and GLaMM proved more effective in spatio-temporal reasoning compared to traditional methods. In the realm of text-to-video generation, commercial models such as Hailuo and Kling 1.5 performed well, though pre-trained models still face challenges due to the limited diversity of underwater content in existing training datasets.
Also Read:
- X-SAM: Advancing Image Segmentation with Unified Multimodal AI
- Pinpointing Events in Videos: A New Approach to Weakly-Supervised Audio-Visual Localization
The introduction of MSC marks a significant step forward in marine video understanding. By providing a large-scale, real-world dataset with detailed annotations, including object segmentation masks and clip-level captions, this work offers a vital resource for researchers. It is expected to facilitate advancements in marine video analysis, contributing to both scientific understanding and conservation efforts. For more details, you can refer to the original research paper.
