TLDR: GS-MoE is a new framework for weakly-supervised video anomaly detection. It uses specialized ‘expert’ AI models for different anomaly types, guided by a ‘Temporal Gaussian Splatting’ technique that helps the model learn from the full duration of an anomaly, not just its most obvious moments. This approach significantly improves detection of complex real-world anomalies like shoplifting and arson, achieving state-of-the-art performance on major datasets.
Detecting unusual events in surveillance videos, known as Video Anomaly Detection (VAD), is a critical yet challenging task. The difficulty arises from the sheer variety of anomalous events and the limited availability of precisely labeled data. Often, models are trained under a ‘Weakly-Supervised’ paradigm (WSVAD), meaning they only receive video-level labels (e.g., ‘this video contains an anomaly’) rather than specific frame-by-frame annotations of when and where the anomaly occurs.
Current state-of-the-art VAD models, while effective for simple anomalies like explosions, often struggle with more complex, real-world events such as shoplifting or subtle assaults. This limitation stems from two main issues: first, these models typically use a single, shared architecture to process all anomaly categories, overlooking the unique features of each type; and second, the weak supervision signal lacks the precise temporal information needed to accurately pinpoint nuanced anomalous patterns that might blend with normal activities.
To address these significant challenges, researchers have introduced a novel framework called Gaussian Splatting-guided Mixture of Experts (GS-MoE). This innovative approach employs a collection of specialized ‘expert’ models, each designed to excel at identifying a particular type of anomaly. These experts are guided by a unique ‘Temporal Gaussian Splatting’ (TGS) loss, which helps the model leverage temporal consistency and enhance the weak supervision signal it receives.
Temporal Gaussian Splatting: Learning the Full Anomaly
Traditional weakly-supervised methods often focus only on the ‘most abnormal’ snippets within a video, potentially overlooking the full duration and subtle aspects of an anomalous event. The Temporal Gaussian Splatting (TGS) technique in GS-MoE offers a new way to optimize this process. Instead of just focusing on the highest-scoring moments, TGS models anomalies as Gaussian distributions. It identifies ‘peaks’ in the abnormal scores predicted by the model and then ‘splats’ Gaussian kernels across the temporal width of these detected peaks. This allows the model to learn a more complete and accurate representation of an anomalous event over time, including snippets that might have lower abnormal scores but are still part of the anomaly.
This method helps in generating more precise ‘pseudo-labels’ for training, providing a target abnormal score for each snippet in the video. This is a significant improvement over previous methods that only pushed a few top snippets towards an ‘abnormal’ score, enabling the model to learn from the entirety of an anomalous event.
Mixture of Experts: Specialized Anomaly Detection
The GS-MoE framework incorporates a sophisticated Mixture-of-Experts (MoE) architecture, which operates in three stages to tackle anomaly diversity:
The first stage involves **Enhanced Temporal-Spatial Feature Extraction**. It uses a combination of general video feature extractors (like I3D) and specialized anomaly detection feature extractors (UR-DMU). These extractors provide rich, informative features that capture both general video dynamics and fine-grained temporal patterns crucial for distinguishing normal from abnormal events.
The second stage is **Class-Specific Anomaly Detection with Expert Models**. This is where the specialization happens. Multiple expert models are introduced, each trained specifically to identify a particular type of anomaly. Each expert, composed of a transformer block and a multi-layer perceptron, learns the unique attributes of its assigned anomaly class. This fine-grained specialization allows the framework to detect subtle or complex anomalies that generic models might miss.
Finally, the third stage features **Collaborative Integration with the Gate Model**. The scores from all the individual expert models are fed into a ‘gate’ model. This gate acts as an intelligent integrator, refining the class-specific anomaly insights and comparing them with the more general anomaly features from the initial feature extraction stage. It uses a bi-directional cross-attention mechanism to understand the correlations and contrasts between expert predictions, ultimately producing a unified and robust final anomaly score.
Also Read:
- Advancing Medical Image Classification with Limited Data: A New Semi-Supervised Learning Framework
- InfiGUI-G1: Boosting AI’s Understanding of User Interfaces with Adaptive Learning
Impressive Performance and Future Outlook
The GS-MoE model has demonstrated state-of-the-art performance across multiple challenging datasets, including UCF-Crime, XD-Violence, and MSAD. On the UCF-Crime dataset, it achieved an impressive 91.58% AUC, significantly outperforming previous leading models. Notably, its performance on abnormal videos alone (AUC A) showed a remarkable 13.63% improvement, highlighting its effectiveness in detecting complex, real-world anomalies. The model also showed significant performance boosts for complex categories like ‘Arson’, ‘Assault’, ‘Fighting’, ‘Stealing’, and ‘Burglary’.
Ablation studies confirmed the critical impact of each component: the Temporal Gaussian Splatting loss, the class-specific experts, and the integrating gate model all contribute significantly to the overall performance. The research also explored the use of cluster-based experts, demonstrating the framework’s adaptability to real-world scenarios where anomaly classes might not be predefined.
By combining the power of specialized expert models with a novel temporal guidance mechanism, GS-MoE sets a new benchmark for weakly-supervised video anomaly detection. The researchers aim to further enhance this framework by leveraging large language models (LLMs) to provide even greater explainability for anomaly classes in the future. You can find the full research paper here.
