TLDR: ViTCoT is a new method that improves how Large Language Models understand videos by integrating key visual information directly into their step-by-step reasoning process, mimicking human cognition. It introduces a new benchmark (ViTIB) and shows significant performance gains across various models compared to text-only reasoning, even activating more neural pathways in the models.
In the rapidly evolving field of artificial intelligence, understanding video content is crucial for advancements in areas like autonomous driving and embodied AI. Large Language Models (LLMs), especially those using Chain-of-Thought (CoT) reasoning, have significantly improved video reasoning. However, a key limitation has been their reliance primarily on textual information, often overlooking the rich visual details within videos—a stark contrast to how humans naturally re-examine visual cues during reasoning.
Addressing this gap, researchers have introduced a groundbreaking new paradigm called Video-Text Interleaved Chain-of-Thought (ViTCoT). This innovative approach aims to make video reasoning more intuitive and aligned with human cognitive processes by actively integrating visual information into the reasoning steps.
The Core Idea: Interleaving Video and Text
ViTCoT’s central innovation lies in its ability to interleave key video segments directly within the textual reasoning process. Unlike traditional methods that process video and text separately and then reason solely on text, ViTCoT allows Multimodal Large Language Models (MLLMs) to “re-examine” visual content as they think step-by-step. This is similar to how a human might pause to re-watch a specific part of a video to clarify a detail while trying to understand a complex scene.
Building the Foundation: The Video-Text Interleaved Benchmark (ViTIB)
To enable and evaluate this new reasoning paradigm, the researchers first constructed a novel dataset called the Video-Text Interleaved Benchmark (ViTIB). This benchmark is meticulously created to integrate video frames with corresponding text, simulating human-like video comprehension. It was built by first automatically removing incomplete data, then using MLLMs (specifically Gemini-2.0-Flash) to identify and extract “key-frames” most relevant to a reasoning task. These key-frames are then assembled into “key-videos.”
A rigorous human recheck process ensures the quality of ViTIB. Three independent reviewers score each data entry, and only those scoring above 80 (out of 100) are retained. If scores are low, reviewers re-discuss and re-select key-frames, ensuring that each key-video frame genuinely supports reaching the correct answer. This meticulous process resulted in a high-quality dataset with an average score of 83.6.
ViTIB covers 14 diverse categories of everyday scenarios, containing 1,382 videos and over 5,000 key-frames. Analysis showed that the semantic features of these key-frames largely overlap with and encompass the textual reasoning content, confirming their relevance and effectiveness in representing the essence of the reasoning process.
How ViTCoT Works: A Two-Stage Reasoning Process
The ViTCoT paradigm operates in two main stages:
- Initial Text Reasoning: In the first stage, the MLLM receives the original video, a question, and options. It is instructed to generate an “Initial Reasoning” process based on this input, without directly providing the final answer. This step establishes a preliminary understanding and reasoning framework.
- Video-Text Interleaved Reasoning: Once the initial reasoning is generated, the “Key-Video” (extracted from the original video) is integrated directly into this initial reasoning. Both the original video and the key-video embedded within the initial reasoning are then provided as context to the MLLM. This allows the model to generate a “Final Reasoning” process and the answer by combining its textual understanding with the relevant visual context from the key-video. This integration enables a more accurate and human-like solution.
Impressive Performance Gains
Extensive experiments demonstrated that ViTCoT significantly boosts performance compared to traditional text-only CoT paradigms. Across various MLLMs, including open-source models like Qwen2.5-VL-3B-Instruct, Qwen2.5-VL-7B-Instruct, VideoLLaMA3-7B, Intern2.5-VL-8B, and the closed-source Gemini-2.0-Flash, methods incorporating Video-Text Interleaved Reasoning consistently outperformed their non-interleaved counterparts. On average, these methods showed a significant improvement of 3.5%.
Notably, on the Qwen2.5-VL-7B-Instruct model, video-text interleaved methods achieved an average improvement of 5.4% over the vanilla reasoning approach. Even when both the original video and the key-video were provided to vanilla reasoning methods, ViTCoT still outperformed them by an average of 2.8%, highlighting the inherent advantage of the interleaved paradigm itself.
Furthermore, even when using “non-Oracle” key-videos (i.e., rough key-videos automatically extracted by CLIP based on initial reasoning), ViTCoT still delivered better performance, with an average improvement of 1.7% over vanilla methods. This confirms that the performance gains are not solely due to perfectly selected key-videos but are a fundamental benefit of the interleaved reasoning approach.
The research also revealed that ViTCoT activates more neuron values within MLLMs, suggesting a deeper and more intensive engagement of the model’s internal mechanisms. This indicates that the ViT paradigm promotes more complex and refined reasoning processes.
Also Read:
- Unlocking Reliable AI Reasoning Through Hidden Cognitive Signals
- Advancing Multimodal AI: A New Model for Unified General and Spatial Understanding
A Step Towards More Human-Like AI
The introduction of ViTCoT marks a significant step forward in video understanding for large language models. By integrating visual information directly into the reasoning process, it enables MLLMs to mimic human cognition more closely, leading to more accurate, intuitive, and effective video comprehension. This work not only provides a novel reasoning paradigm but also a crucial benchmark for future research in multimodal AI. For more technical details, you can refer to the full research paper.
