TLDR: EgoPrune is a novel training-free method designed to efficiently process first-person (egomotion) videos for embodied AI. It addresses the high computational cost of vision-language models by intelligently pruning redundant visual tokens. The method employs a keyframe selector, Perspective-Aware Redundancy Filtering (PARF) for geometric alignment and redundancy removal, and a Maximal Marginal Relevance (MMR) token selector that balances query relevance and visual diversity. EgoPrune significantly reduces FLOPs, memory, and latency while maintaining high accuracy, proving its effectiveness for real-world deployment on edge devices.
Egomotion videos, which are first-person recordings from a moving agent, are crucial visual inputs for embodied AI. These videos capture the world as an AI agent perceives it, enabling it to understand its surroundings and make decisions. However, processing these long, continuous video streams efficiently has been a significant challenge for advanced vision-language models (VLMs).
Recent breakthroughs in VLMs have brought powerful multimodal reasoning capabilities, but their computational demands are often too high for the extensive, redundant video data. Traditional token pruning methods, typically designed for third-person videos, struggle with egomotion videos because they don’t account for the continuous viewpoint changes and motion constraints inherent in first-person perspectives. This often leads to the erroneous removal of essential visual information.
To address this, researchers have introduced EgoPrune, a novel training-free token pruning method specifically designed for egomotion video reasoning. EgoPrune aims to make egomotion video processing more efficient for real-world deployment without sacrificing accuracy.
How EgoPrune Works
EgoPrune operates through three key components:
1. Keyframe Selector: Adapted from a method called EmbodiedR, this component efficiently samples frames over time. It intelligently picks out keyframes, ensuring that important temporal information is captured without processing every single frame.
2. Perspective-Aware Redundancy Filtering (PARF): This is a crucial innovation for egomotion videos. Unlike methods that assume static camera views, PARF understands that the viewpoint is constantly shifting. It uses perspective transformations to align visual tokens between consecutive frames. By doing so, it can accurately identify and remove tokens that are truly redundant, even as the camera moves. This geometric alignment is vital for maintaining spatial completeness.
3. Maximal Marginal Relevance (MMR)-based Token Selector: After the initial filtering, this component further refines the selection of visual tokens. It balances two important criteria: how relevant a token is to the user’s text query and how diverse the selected tokens are within a single frame. This ensures that the retained tokens are not only informative for the task at hand but also provide a broad visual representation of the scene, preventing the loss of critical context.
Performance and Efficiency
Extensive experiments on two egomotion video benchmarks, VSI-Bench (for indoor reasoning) and UrbanVideo-Bench (for urban aerial scenarios), have shown that EgoPrune consistently outperforms existing training-free methods. It maintains over 99% of task accuracy while significantly reducing computational costs, memory usage, and processing latency. For instance, on VSI-Bench, EgoPrune often matched or even exceeded the accuracy of models using full token sets, even when pruning 50% or 70% of the visual tokens.
The efficiency gains are particularly notable for longer video inputs, where EgoPrune demonstrates lower computational cost, reduced memory usage, and smoother scaling in latency. This makes it highly suitable for practical applications.
Also Read:
- Gaze-Guided Robots: Enhancing Efficiency and Robustness with Human-Inspired Vision
- Smart Routing for Video Search: A New Approach to Efficiency
Real-World Deployment
A significant aspect of EgoPrune’s validation is its successful deployment on a Jetson Orin NX 16GB edge device. This demonstrates its real-world efficiency and suitability for on-device egomotion video reasoning, which is critical for embodied agents like those used in UAV navigation or mobile robotics. The method effectively reduces end-to-end latency and peak GPU memory usage, allowing for faster processing and stable operation alongside other onboard systems.
In conclusion, EgoPrune offers a robust and lightweight solution for efficient egomotion video reasoning in embodied AI agents. By intelligently pruning redundant visual information while preserving task-relevant details, it paves the way for more practical and deployable embodied AI systems. You can read the full research paper here.
