TLDR: A new framework, SG-VLM, improves video question answering by integrating symbolic scene graphs into existing Vision Language Models (VLMs). It generates, selects, and uses these structured object-relation representations to enhance causal and temporal reasoning, offering more interpretable and accurate answers, especially for specific question types, though the quality of scene graph extraction remains a key factor.
Understanding and answering questions about videos is a complex task for artificial intelligence. While advanced Vision Language Models (VLMs) have shown impressive capabilities, they often struggle with truly understanding the “why” and “when” behind actions in videos, relying instead on superficial patterns. This can lead to models making educated guesses rather than truly reasoning about the events unfolding on screen.
A new research paper, “Bridging Vision Language Models and Symbolic Grounding for Video Question Answering”, by Haodi Ma, Vyom Pathak, and Daisy Zhe Wang from the University of Florida, introduces an innovative approach called SG-VLM. This framework aims to enhance the reasoning abilities of existing VLMs by integrating “symbolic scene graphs” as an intermediate step. Think of scene graphs as structured descriptions of what’s happening in a video frame – identifying objects and their relationships (e.g., “orange cat watching tabby cat,” “tabby cat eating food”).
How SG-VLM Works
SG-VLM is designed to be modular and works with pre-trained VLMs without needing extensive re-training. It operates in three main stages:
1. Scene Graph Generation: Instead of relying on separate, complex systems to create scene graphs, SG-VLM cleverly uses the VLM itself to generate these structured descriptions from sampled video frames. It identifies key objects and their spatial (e.g., “next to”) and action-centric (e.g., “watching”) relationships. It also tracks actions over time to understand continuity.
2. Scene Graph Selection: Not all information in a video is relevant to a specific question. This stage acts as a filter, identifying only the scene graphs from frames that are most pertinent to the question asked. For example, if the question is about why one cat is watching another eat, it will focus on frames where these specific interactions are happening, ignoring background details.
3. Grounded Answer Generation: Finally, the selected, question-relevant scene graphs are combined with the original video context and fed back into the VLM. This enriched input helps the VLM generate more accurate, interpretable, and causally sound answers. For instance, it might help the model deduce that a cat is “waiting for its turn” based on the observed interactions.
Key Findings and Insights
The researchers tested SG-VLM across three diverse video question answering benchmarks: NExT-QA, iVQA, and ActivityNet-QA, using different VLM backbones like QwenVL and InternVL. The results showed that SG-VLM consistently improved causal and temporal reasoning compared to prior methods. While the gains over already very strong VLMs were sometimes modest, the framework demonstrated a clear benefit in providing structured grounding.
An important discovery was that the way scene graphs are integrated matters significantly. Simply using all generated scene graphs (Full-SG) often introduced noise. Instead, selecting only the question-relevant graphs (FrameSel-SG) or even just summarizing objects without detailed relations (Summary-SG) often yielded better performance. This highlights that quality and relevance of symbolic information are more crucial than sheer quantity.
The study also revealed that symbolic grounding is particularly beneficial for certain types of questions. For example, object-only graphs helped with descriptive questions, while frame selection improved counting and predicting what happens next. However, for complex “Causal Why” questions where VLMs already perform well, adding symbolic input could sometimes introduce redundancy or noise.
Also Read:
- Advancing Robot Perception: Incremental 3D Scene Understanding with Heterogeneous Graphs
- Understanding How AI Connects Words to Images
Future Directions
This work underscores the potential of combining the holistic understanding of VLMs with the structured, interpretable reasoning provided by symbolic scene graphs. While challenges remain, particularly in improving the accuracy of relation extraction and dynamically deciding when and how to best apply symbolic grounding, SG-VLM offers a promising path towards more faithful and explainable video understanding systems.
