TLDR: Researchers Vishakha Lall and Yisi Liu introduce ‘Eyes on Target,’ a novel depth-aware and gaze-guided object detection framework for egocentric videos. This system integrates human gaze data (position, depth, pupil diameter, direction) into a Vision Transformer’s attention mechanism, biasing it towards human-attended regions. The approach significantly improves object detection and classification accuracy over traditional methods and gaze-agnostic baselines, especially in complex egocentric scenarios like maritime simulations. The paper also proposes a ‘gaze-aware head importance’ metric for model interpretability.
A new research paper introduces an innovative approach to object detection in first-person videos, leveraging human gaze as a crucial signal. Titled “Eyes on Target: Gaze-Aware Object Detection in Egocentric Video,” this work by Vishakha Lall and Yisi Liu from Singapore Polytechnic proposes a novel framework that significantly enhances how artificial intelligence understands and prioritizes visual information, especially in dynamic, real-world scenarios captured from a person’s perspective.
Traditional object detection systems often treat all parts of a video frame equally, which can be inefficient and less accurate when human attention is a key factor. The researchers argue that human gaze offers rich supervisory signals, indicating where a person is looking and what they are focusing on. This is particularly vital in egocentric videos, which are captured from a wearable camera, offering insights into user attention and intent in fields like virtual reality, augmented reality, robotics, and simulation-based training.
The core of their innovation, named ‘Eyes on Target,’ is a depth-aware and gaze-guided object detection framework built upon the Vision Transformer (ViT) architecture. This framework injects features derived from human gaze – such as gaze position, depth, pupil diameter, and direction – directly into the attention mechanism of the ViT. By doing so, the model’s spatial feature selection is effectively biased towards regions that a human viewer is attending to, rather than treating all regions uniformly.
Imagine a maritime training simulator where a captain is navigating a vessel. Tracking where the captain’s eyes are focused on the control panel or radar is critical for assessing their performance. The ‘Eyes on Target’ model is designed to excel in such environments. It processes images by dividing them into patches, and then, using the gaze information, it modifies the attention scores between these patches. This ensures that patches closer to the gaze point and aligned with the gaze direction receive increased attention weights, making the model focus more on these human-prioritized areas.
Beyond just guiding attention, the model also refines the bounding boxes around detected objects. This refinement uses a dynamic scaling factor that considers the average attention score within a region, as well as the pupil diameter (an indicator of cognitive attention) and object depth. This allows the model to generate tighter, more context-aware bounding boxes around areas of visual focus, adapting to how a human might perceive and interact with objects at different distances and levels of engagement.
The researchers validated their method on several datasets, including a custom Egocentric Maritime Simulator Dataset, the Ego Motion dataset, and the Ego-CH-Gaze dataset. Their extensive experiments and ablation studies demonstrated consistent gains in detection accuracy over gaze-agnostic baselines. For instance, on the Egocentric Maritime Simulator Dataset, their model achieved significantly higher classification accuracy compared to existing methods, showcasing the utility of integrating gaze as contextual guidance.
To further understand how gaze influences the model, the team also introduced a novel ‘gaze-aware attention head importance’ metric. This metric quantifies the impact of gaze data on different attention heads within the transformer, revealing how gaze cues modulate the model’s internal attention dynamics. This provides valuable interpretability, showing which parts of the model become more sensitive to human visual patterns when gaze information is integrated.
Also Read:
- Bridging Modalities: UniSOT’s Unified Approach to Object Tracking
- Modeling Realistic Pedestrian-Driver Interactions with Human-Like Constraints
In conclusion, the ‘Eyes on Target’ framework represents a significant step forward in object detection for egocentric videos. By deeply integrating human gaze cues, including gaze point, depth, pupil dilation, and direction, into Vision Transformer architectures, the model achieves enhanced performance in tasks where human context and intent are crucial. This research not only improves object detection but also opens doors for future work in explainable AI and human-in-the-loop learning, particularly in safety-critical and simulation-based training environments. You can read the full research paper here.
