TLDR: A new study introduces a framework and three datasets to test how well AI action recognition models generalize high-level motions to unseen contexts (e.g., “punching a person” vs. “punching a bag”). Findings show significant performance drops in novel scenarios, especially for fine-grained actions. Models over-rely on visual cues and background, and larger models don’t always improve temporal understanding. A proposed disentanglement strategy helps improve generalization by focusing on core motion.
A new study delves into a critical challenge for artificial intelligence models that recognize actions in videos: their ability to transfer learned motion concepts to new, unseen situations. While current action recognition models are quite good at identifying activities they’ve been trained on, a key question remains: can they understand a broad action like “punching” when it appears in a slightly different, unfamiliar context, such as “punching a person” instead of a “punching bag”?
Researchers Raiyaan Abdullah, Jared Claypoole, Michael Cogswell, Ajay Divakaran, and Yogesh Rawat explored this question in their paper, “Punching Bag vs. Punching Person: Motion Transferability in Videos.” The study highlights that even state-of-the-art models often fail to transfer their understanding of high-level, general motions to contexts they haven’t encountered during training. This limitation is particularly evident with fine-grained actions, where models struggle to distinguish subtle differences in novel scenarios.
Understanding the Challenge of Motion Transfer
The core issue lies in how these AI models learn. They tend to develop biases based on the specific contexts seen in their training data. For instance, a model trained extensively on videos of “punching a bag” might struggle to recognize the same “punching” action when the object being punched is a person, even though the fundamental motion is similar. This is referred to as “fine-class bias,” where models get stuck on specific details rather than generalizing the core action.
To systematically investigate this, the researchers introduced a novel motion transferability framework and three new benchmark datasets:
- Syn-TA: A synthetic dataset created using 3D objects in Blender. This controlled environment allowed researchers to precisely vary actors and contexts, providing a clear way to test a model’s ability to generalize high-level motion concepts without real-world clutter.
- Kinetics400-TA: Adapted from the well-known Kinetics400 dataset, this real-world dataset focuses on actions where spatial cues (objects, actors, scene details) are often dominant in defining the action.
- Something-Something-v2-TA (SSv2-TA): Derived from Something-Something-v2, this dataset emphasizes actions where understanding temporal dynamics (the sequence and timing of movements) is crucial for correct recognition.
Each dataset was structured with a hierarchy of “coarse” (high-level) classes, like “Pushing,” and “fine” (lower-level, context-specific) classes, such as “Pushing something from left to right.” The datasets were split to ensure that models were tested on both “known” contexts (seen during training) and “unknown” contexts (unseen variations of the same coarse action).
Key Findings from Extensive Evaluation
The study evaluated 13 state-of-the-art unimodal (processing one type of data, like video) and multimodal (processing multiple types, like video and text) models. The results consistently showed a significant drop in performance when models had to recognize high-level actions in unknown contexts. Here are some key insights:
- Generalization Gap: All models, regardless of their architecture, showed a substantial decrease in accuracy when moving from known to unknown scenarios.
- Fine Motions are Harder: Multimodal models, which can classify unseen fine-grained motions, performed even worse on these compared to coarse motions in unknown contexts, indicating greater difficulty with subtle distinctions.
- Synthetic Data’s Challenge: Surprisingly, the controlled Syn-TA dataset proved more challenging for most models than the real-world Kinetics400-TA. This suggests that models heavily rely on familiar visual cues from real-world scenes, and when these are absent, their core motion understanding is tested more rigorously.
- Model Size vs. Understanding: While larger models generally performed better in tasks relying on spatial cues (like in K400-TA), their increased size didn’t necessarily translate to better understanding of temporal information (as seen in SSv2-TA).
- Background Bias: Experiments with Syn-TA using plain backgrounds instead of realistic ones showed improved performance. This indicates that models often over-rely on background textures and information, hindering their ability to generalize motion accurately.
Also Read:
- Enhancing Video Anomaly Detection with AI-Generated Training Data
- Bridging the Cognitive Divide: Why AI’s Goals Differ from Human Intentions
A Step Towards Better Motion Understanding
The researchers also proposed a “disentanglement strategy” to improve motion transferability. This involves training models to predict both high-level coarse concepts and fine-grained details simultaneously, using separate branches in the model’s architecture. By integrating fine-context features into the coarse branch, the model is encouraged to focus more on the motion itself rather than getting sidetracked by irrelevant contextual details. This approach showed promising results, particularly in temporal datasets like Syn-TA and SSv2-TA.
This research establishes a crucial benchmark for evaluating how well AI models can truly understand and transfer motion concepts across diverse contexts. It highlights the need for future models to move beyond context-dependent biases and develop a more robust, generalized understanding of actions. For more technical details, you can refer to the full research paper available here.
