TLDR: A new benchmark called VIDEONORMS has been introduced to assess the cultural awareness of Video Large Language Models (VideoLLMs). It features over 1000 video clips from US and Chinese cultures, annotated with socio-cultural norms, adherence/violation labels, and verbal/non-verbal evidence. The study found that VideoLLMs struggle more with detecting norm violations, understanding Chinese culture compared to US culture, extracting non-verbal cues, and performing in formal contexts, emphasizing the need for culturally-grounded AI training.
As artificial intelligence systems, particularly Video Large Language Models (VideoLLMs), become increasingly integrated into global applications, their ability to understand and navigate diverse cultural contexts is paramount. However, the cultural competence of these models has not received as much attention as other areas like object recognition or temporal reasoning. To address this critical gap, researchers have introduced a new benchmark called VIDEONORMS.
The VIDEONORMS benchmark is a comprehensive dataset designed to evaluate how well VideoLLMs understand socio-cultural norms. It comprises over 1000 pairs of video clips and associated norms, drawing from both US and Chinese cultures. Each pair is meticulously annotated with details such as the specific socio-cultural norm, whether the norm is adhered to or violated, and concrete verbal and non-verbal evidence supporting these labels. This rich annotation allows for a deep assessment of a model’s cultural awareness.
Building such a detailed dataset was a significant undertaking, accomplished through an innovative human-AI collaboration framework. In the first stage, a “teacher model” – a powerful VideoLLM – was prompted using principles from speech act theory to generate initial candidate annotations. These annotations included the norm category, the specific socio-cultural norm, an adherence or violation label, and relevant verbal and non-verbal evidence. Following this, a team of trained human experts, each with a relevant cultural background, meticulously reviewed and corrected these candidate annotations, ensuring accuracy and cultural nuance.
The benchmark proposes three progressively challenging tasks to evaluate VideoLLMs:
Task 1: Predicting Adherence or Violation
Given a video segment, its transcript, a norm category, and a specific norm, the model must determine if the observed behavior adheres to or violates that norm. This is a binary classification task.
Task 2: Predicting Adherence/Violation and Extracting Evidence
Building on the first task, models must not only predict adherence or violation but also generate two types of evidence: verbal evidence (referencing spoken content) and non-verbal evidence (referencing visual cues like gaze, gestures, or facial expressions). This task assesses the model’s ability to justify its predictions with concrete details from the video.
Also Read:
- Assessing Advanced Video Reasoning in Multimodal AI for Science
- EvalMORAAL: Unpacking Cultural Morality and Bias in Large Language Models
Task 3: Predicting a Cultural Norm
In this task, given a video segment, its transcript, and a norm category, the model is challenged to generate the specific cultural norm that best captures the exhibited behavior in the video.
The researchers benchmarked a variety of open-weight VideoLLMs on VIDEONORMS, revealing several consistent trends. A significant finding was that models generally performed worse when detecting norm violations compared to identifying norm adherence. Furthermore, a clear cultural disparity emerged: models consistently performed worse with Chinese cultural contexts compared to US culture, highlighting a potential “WEIRD” (Western, Educated, Industrialized, Rich, and Democratic) bias in current models. The study also found that models struggled more with providing non-verbal evidence than verbal evidence, suggesting a weakness in interpreting subtle visual social cues. Additionally, models had difficulty precisely identifying the exact norm corresponding to a speech act. Interestingly, unlike human annotators who showed higher agreement in formal settings, the models performed worse in formal, non-humorous contexts.
These findings underscore a crucial need for culturally-grounded training for video language models. The VIDEONORMS benchmark and its construction framework offer a foundational step towards addressing this gap, providing a valuable tool for developing more culturally aware AI systems. For more in-depth information, you can read the full research paper here.
