TLDR: A novel two-stage training framework, VTG-R1, significantly improves Video Temporal Grounding (VTG) by combining supervised fine-tuning (SFT) with reinforcement learning (RL). The SFT stage uses high-quality ‘cold start’ data for initial model training, followed by a difficulty-controlled RL stage that enhances temporal localization and reasoning. This approach consistently outperforms existing models, demonstrating the crucial roles of both high-quality initial data and strategic RL training.
Video Temporal Grounding (VTG) is a crucial technology that helps pinpoint specific moments within videos based on natural language queries. Imagine trying to find a particular action in a long video without having to watch the whole thing – that’s what VTG aims to achieve. This capability is vital for many applications, from intelligent video retrieval on social media to automated event monitoring in industrial settings.
Despite advancements, especially with large vision-language models (LVLMs), current VTG approaches often struggle with accurately understanding temporal aspects and generalizing to new, unseen scenarios. Many existing models rely heavily on a technique called supervised fine-tuning (SFT), which, while effective, has inherent limitations in capturing precise temporal awareness.
To overcome these challenges, researchers have introduced a novel two-stage training framework that integrates supervised fine-tuning with reinforcement learning (RL). This new approach, named VTG-R1, is designed to significantly improve both the accuracy and robustness of VTG models.
The Two-Stage Training Framework
The framework begins with a supervised fine-tuning (SFT) stage, often referred to as the “cold start” phase. In this stage, the model is initialized using high-quality, carefully curated data. This initial training provides the model with a strong foundation in understanding multimodal information and structured reasoning, preparing it for the next phase.
Following the SFT, the model enters the reinforcement learning (RL) stage. This is where the model’s temporal localization and reasoning abilities are further enhanced. The RL process is guided by a composite reward function that encourages both accurate temporal segment prediction (measured by Intersection-over-Union, or IoU) and structured reasoning in the model’s output. The training uses a method called Group Relative Policy Optimization (GRPO), which helps the model learn to assign higher probabilities to better responses without needing a separate critic model, thus reducing computational complexity.
Also Read:
- Advancing Long-Form Video Analysis with Controllable Hybrid Captioning
- VideoMind: A New Dataset for Advanced Video Comprehension
Key Findings and Impact
Extensive experiments conducted on multiple VTG benchmarks, including NExTGQA, RexTime, and Charades-STA, demonstrate that VTG-R1 consistently outperforms existing SFT-based methods. This highlights the significant impact of integrating reinforcement learning, especially in challenging and open-domain scenarios.
A critical insight from this research is the importance of high-quality cold start data. Models initialized with superior data in the SFT phase converge to higher performance scores in the RL phase, indicating that a strong initial foundation unlocks the model’s full potential. Furthermore, controlling the difficulty of the data used in the RL training stage is also crucial. Filtering out overly difficult or confusing samples helps the model learn more effectively, particularly if it hasn’t had a robust cold start.
This work not only presents a powerful new framework for video temporal grounding but also emphasizes the importance of both high-quality initial data and carefully controlled reinforcement learning for developing robust reasoning capabilities that generalize well. To foster further research and adoption, all intermediate datasets, models, and code are being released as open-source resources. You can find the full research paper here: Research Paper.
