TLDR: This research investigates “temporal label noise” in multimodal hateful video classification, where videos labeled as hateful often contain non-hateful segments. By trimming videos to isolate explicitly hateful parts, the study reveals that current video-level annotations introduce significant noise, leading to semantic overlap and reduced model performance. Experiments show that using precise, segment-level annotations drastically improves detection accuracy, highlighting the need for more temporally aware models and annotation strategies for hate speech detection.
The rapid spread of online multimedia content, especially videos, has unfortunately amplified the propagation of hate speech, posing significant challenges for society and regulators. While advancements have been made in detecting hateful videos using various modalities like text, audio, and visuals, most existing methods rely on broad, video-level annotations. This means an entire video is labeled as either hateful or non-hateful, even if only a small portion contains hate speech. This approach, however, introduces a substantial problem: ‘temporal label noise’.
The Hidden Problem of Temporal Label Noise
Imagine a video that is two minutes long and labeled as ‘hateful’. It might only contain a few seconds of actual hate speech, with the rest being irrelevant or non-hateful content. This discrepancy, where non-hateful segments are included within a video broadly labeled as hateful, is what researchers call temporal label noise. This study, titled “Revealing Temporal Label Noise in Multimodal Hateful Video Classification” by Shuonan Yang, Tailin Chen, Rahul Singh, Jiangbei Yue, Jianbo Jiao, and Zeyu Fu, delves into this critical issue. The authors highlight that in datasets like HateMM, hateful videos average 2.56 minutes, but the hateful segments within them average only 1.71 minutes, meaning about 33% of the content in a ‘hateful’ video is actually non-hateful. A similar pattern was observed in the MultiHateClip-English dataset.
A Fine-Grained Approach to Understanding Hate
To address this, the researchers adopted a fine-grained approach. They utilized annotated timestamps from the HateMM and MultiHateClip-English datasets to precisely trim hateful videos, isolating only the explicitly hateful segments. They also identified ‘trimmed non-hate segments’ from within these same hateful videos, which occurred before or after the hateful spans. This meticulous preprocessing allowed for a controlled analysis of the semantic boundaries and the impact of label noise.
Unpacking the Data: Lexical, Audio, and Visual Insights
The study conducted an extensive exploratory analysis across different modalities:
-
Lexical Analysis: By examining the transcribed text, they found significant differences in emotionally charged categories between trimmed hate and non-hate segments. Hate segments, especially from the BitChute platform, showed higher scores for terms related to ‘negative emotion’, ‘violence’, and ‘hate’. However, the presence of some offensive terms in both hate and non-hate segments indicated a semantic overlap, suggesting that the boundaries are not always clear-cut.
-
Audio Analysis: While trimmed hate segments generally exhibited higher average energy levels (suggesting greater vocal intensity), there were minimal sustained differences in spectral characteristics (Zero Crossing Rate) between hate and non-hate segments. This acoustic similarity suggests that hate speech often involves gradual semantic transitions rather than abrupt shifts, making precise temporal isolation challenging.
-
Visual Analysis: Object detection, specifically for ‘persons’, showed negligible differences between trimmed hate and non-hate segments. This visual homogeneity implies that hateful content often occurs within continuous visual contexts, further complicating the distinction based on visual cues alone.
-
Semantic Space Visualization: Using BERT embeddings and UMAP for visualization, the researchers clearly demonstrated the problem. Original video-level annotations showed substantial semantic overlap between hate and non-hate content. While trimming improved separation, trimmed hate and trimmed non-hate segments from the *same* video still showed considerable overlap. This indicates that content labeled as non-hateful within a hateful video is semantically distinct from genuinely non-hateful content, confirming that video-level labels introduce systematic contamination.
Quantifying the Impact: Experiments and Results
To systematically assess the impact of temporal label noise, the study designed four experimental configurations:
-
Coarse Video-level Detection: The baseline, using conventional video-level annotations.
-
Noisy-to-Clean Generalization: Training on noisy full videos and testing on clean, trimmed hateful content.
-
Clean-to-Noisy Generalization: Training on clean, trimmed hateful content and testing on noisy full videos.
-
Clean Segment-Level Detection: The ideal scenario, training and testing only on precisely trimmed hateful segments and non-hateful videos.
The results were striking. Clean segment-level detection significantly outperformed coarse video-level approaches, achieving Macro F1 score improvements of 19.34% on HateMM and 30.45% on MultiHateClip-English. This clearly quantifies the performance cost of label noise. Furthermore, models trained on noisy data struggled to identify trimmed hateful content, and models trained on clean data could not effectively handle the temporal complexity in contaminated videos. This demonstrates that label noise fundamentally alters how models make decisions, rather than just introducing random errors.
Also Read:
- Pinpointing Events in Videos: A New Approach to Weakly-Supervised Audio-Visual Localization
- Defending Vision-Language Models: Strategies for Robustness Against Adversarial Attacks
The Path Forward: Temporally Aware Models
This research provides compelling evidence that current video-level annotations are inadequate for capturing the localized and context-dependent nature of hate speech. The significant performance gains achieved with clean, segment-level data underscore the urgent need for developing ‘temporally aware’ detection models and improved annotation strategies. Such advancements would lead to more robust and interpretable hate speech detection systems, crucial for effective content moderation in our increasingly multimedia-rich online world. You can read the full research paper here.
