TLDR: A new research paper introduces SceneQA, a video question-answering task focusing on scene-level understanding in long videos, and the LVSQA dataset to support it. They propose SLFG (Scene-Localized Frame Grouping), a plug-and-play method that groups video frames into semantically coherent scenes, significantly improving Multimodal Large Language Models’ ability to answer complex questions about long videos by mimicking human cognitive processes.
Understanding long videos has always been a significant challenge for artificial intelligence, especially for Multimodal Large Language Models (MLLMs). These advanced AI models often struggle with processing the vast amount of information contained in lengthy videos, leading to difficulties in answering questions about specific events or details. The core problem lies in resource limitations, which prevent MLLMs from analyzing every single frame, making it hard to extract truly relevant information efficiently.
Traditional video question-answering (VideoQA) methods typically focus on identifying individual frames that contain core objects. However, this approach doesn’t align well with how humans understand videos or with the demands of real-world applications. Humans don’t analyze every frame; instead, they focus on coherent scenes and the details within them. This discrepancy highlights a critical gap in current AI video understanding.
To bridge this gap, a new research paper introduces an innovative approach called SceneQA, a novel scenario under the video question-answering task. SceneQA emphasizes the perception of details and reasoning abilities based on entire scenes rather than isolated frames. This new task aims to challenge MLLMs to comprehend coherent scenes as a whole, requiring them to answer complex questions that involve contextual reasoning about relationships between multiple elements within a scene.
To support the SceneQA task, the researchers developed a new dataset called LVSQA (Long Video Scene-level Question Answering Dataset). This dataset is built upon carefully selected long videos from LVBench, each over 30 minutes in length. It features a new collection of high-quality question-answer pairs designed to promote a fairer evaluation of MLLMs’ scene perception abilities in long videos. The questions in LVSQA focus on detailed visual understanding, making it a challenging and targeted testbed for assessing models’ scene-level localization and fine-grained visual reasoning.
Inspired by human cognition, the paper also proposes a novel method called SLFG (Scene-Localized Frame Grouping). The fundamental idea behind SLFG is to combine individual video frames into semantically coherent “scene frames.” This method leverages scene localization techniques and dynamic frame reassembly mechanisms to significantly improve the understanding capabilities of existing MLLMs in long videos. A key advantage of SLFG is its “plug-and-play” usability, meaning it requires no modifications to the original model architecture, making it easy to integrate with existing MLLMs.
How SLFG Works
The SLFG framework operates in four main stages. First, it performs dense sampling of the video and groups consecutive frames into fixed-granularity frame groups. A Multimodal Large Language Model (MLLM) then generates detailed visual descriptions for each group, capturing objects, actions, and relationships. Second, a Large Language Model (LLM) abstracts these detailed descriptions into more concise, scene-level semantic representations. This step helps reduce noise and redundancy from low-level frame descriptions, creating summaries that reflect the overall semantic structure of a scene.
Third, SLFG uses an open-source text embedding model to calculate the semantic similarity between these video scene descriptions and the question being asked. This helps identify the most relevant scenes where the answer is likely to be found. Finally, a dynamic frame group reorganization module adjusts the structure of the frame groups based on their similarity scores. High-relevance groups are merged, low-relevance groups are discarded, and the time window of key segments is expanded to enhance information density and reasoning effectiveness within the model’s limited context window.
Also Read:
- A New Approach to Understanding Text in Videos: Introducing GAT
- Chain of Questions: Empowering Language Models with Multimodal Curiosity
Experimental Results and Impact
Experimental results demonstrate that SLFG performs exceptionally well across several long video benchmark tests. When applied to base models like LLaVA-OneVision and LLaVA-Video, SLFG achieved significant accuracy improvements. For instance, on the VideoMME w/o long benchmark, LLaVA-OneVision’s performance increased from 43.6% to 46.9%, and LLaVA-Video’s from 47.6% to 49.5%. On the LVSQA dataset, SLFG also showed substantial improvements, highlighting its ability to enhance MLLMs’ capacity to capture detailed visual information in long videos.
The research also compared SLFG with other retrieval methods, concluding that while methods like Adaptive Keyframe Sampling (AKS) are good for “finding things,” SLFG is superior for “understanding the story.” This is because SLFG treats videos as a series of coherent scenes, preserving the context and logic needed for narrative comprehension. Furthermore, SLFG proves to be time-efficient, especially as the number of questions for a single video increases, because its initial processing stages are executed only once and reused.
This work marks a significant step forward in long video understanding, offering a practical and effective method for improving MLLMs’ ability to perceive and reason about complex events within lengthy video content. The code and dataset will be released to the public, fostering further research in this critical area. You can find more details about this research paper here: RESEARCH_PAPER_URL.
