TLDR: StreamMem is a novel method that allows Multimodal Large Language Models (MLLMs) to efficiently process and understand long, streaming videos without running out of memory. It achieves this by intelligently compressing the model’s visual memory (KV cache) in a ‘query-agnostic’ way, meaning it doesn’t need to know the user’s questions in advance. This makes it ideal for real-time video analysis and conversational AI in memory-constrained environments.
Multimodal Large Language Models (MLLMs) have shown impressive capabilities in understanding both visual and language information. However, their ability to efficiently process and understand long videos has been a significant challenge. Traditional methods often struggle with the vast amount of memory and computational power required to store and access the ‘key-value’ (KV) cache, which is essentially the model’s memory of the visual context.
Existing solutions for video compression in MLLMs often fall short. Some require processing the entire video before compression, which isn’t practical for live streaming. Others need to know the specific questions that will be asked in advance, making them unsuitable for real-time, multi-turn conversations where questions are unpredictable.
Introducing StreamMem
A new research paper, titled “StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding,” introduces an innovative solution to these problems. Developed by researchers Yanlai Yang, Zhuokai Zhao, Satya Narayan Shukla, Aashu Singh, Shlok Kumar Mishra, Lizhu Zhang, and Mengye Ren from Meta AI and New York University, StreamMem is a query-agnostic KV cache memory mechanism designed specifically for streaming video understanding. You can find the full paper at https://arxiv.org/pdf/2508.15717.
The core idea behind StreamMem is to process new video frames as they arrive in a continuous stream. It intelligently compresses the KV cache by using attention scores between visual tokens (the model’s representation of visual information) and generic query tokens. This allows StreamMem to maintain a fixed-size KV memory, which is crucial for efficient question answering in situations with limited memory, especially when dealing with very long videos.
How StreamMem Works
StreamMem employs several key components to achieve its efficiency:
-
Input Frame Filtering: Before the MLLM processes them, incoming video clips go through a lightweight filtering step. This step identifies and removes redundant frames, such as those from static scenes or high-frame-rate videos, ensuring that the KV cache isn’t overwhelmed with repetitive information.
-
Query-Agnostic KV Cache Compression: Instead of relying on a specific user query, StreamMem uses the system’s ‘chat template tokens’ as a proxy. These tokens implicitly prompt the MLLM to generate a generic video description, allowing the model to identify and retain informative visual content without knowing the actual question beforehand.
-
Attention-Based Pruning: Based on the attention scores generated by the proxy query, StreamMem selects and preserves the most salient (important) visual tokens, effectively pruning less relevant information.
-
Frame-wise KV Merging: To further compress memory, StreamMem computes a ‘prototype’ key and value representation for each frame. This is done by taking a weighted average of the keys and values based on their importance scores, creating compact yet expressive memory representations.
-
Positional Embedding with YaRN: MLLMs often struggle with very long video sequences because they aren’t extensively trained on such data. StreamMem addresses this by adopting the YaRN context window extension technique, originally for language models, to extend the visual context capacity and preserve positional consistency across streaming segments.
Also Read:
- RynnEC: A Video Multimodal Model for Embodied Cognition
- Adaptive Planning for LLM Agents with Grounded Memory
Performance and Impact
Evaluations across various long video understanding and streaming video question answering benchmarks, including MLVU, EgoSchema, VideoMME, RVS-Ego, and RVS-Movie, demonstrate StreamMem’s effectiveness. It achieves state-of-the-art performance in query-agnostic KV cache compression and is competitive with approaches that require prior knowledge of the query.
The research also includes detailed ablation studies, confirming the individual contributions of each component within the StreamMem framework, such as input frame filtering and KV merging strategies. These findings highlight the benefits of streaming-based KV processing and compression over traditional uniform frame sampling.
StreamMem represents a significant step towards building real-time MLLM agents capable of continuous video understanding in diverse, open-world settings. By enabling scalable memory compression without the need for fine-tuning, it addresses a critical bottleneck in deploying advanced AI models for long-form video analysis.
