TLDR: EventFormer introduces Action-centric Video Event Prediction (AVEP), a new task for predicting future video events and their arguments with complex logic and rich semantics. It comes with a large, structured dataset of 35K videos and 178K events. The EventFormer model uses a node-graph hierarchical attention Transformer and a novel coreference encoding mechanism to capture relationships between events and arguments. It significantly outperforms existing video prediction models and LVLMs, establishing a strong baseline for this challenging task.
Predicting what happens next in a video, especially when it involves complex human actions and interactions, has long been a significant challenge in artificial intelligence. While script event induction has seen success in natural language processing, applying similar concepts to video has been difficult due to the visual modality’s inherent complexities.
Researchers have introduced a groundbreaking new task called **Action-centric Video Event Prediction (AVEP)**. This task goes beyond simple action anticipation or visual reasoning by incorporating more intricate logic and richer semantic information. Unlike previous approaches that might focus on low-level verbs like “run,” AVEP aims to predict higher-level concepts such as “chase” or “discuss,” along with detailed descriptions of the participants involved.
To support this ambitious task, a substantial new dataset has been created, also named AVEP. This dataset is built upon existing video event datasets like VidSitu, Epic-kitchens, and VidEvent, but it offers significantly more fine-grained annotations. It comprises approximately 35,000 annotated videos and over 178,000 video clips of events. The atomic unit in this dataset is a multimodal event argument node, providing a highly structured representation of video events. This structured approach is crucial because traditional visual models, which typically process patches or frames, are not well-suited to capture the complex relationships within and between events.
Introducing EventFormer: A Novel Prediction Model
To tackle the unique challenges of AVEP, the researchers propose **EventFormer**, a novel video event prediction model. EventFormer is a node-graph hierarchical attention-based Transformer designed to understand both the relationships between events and their arguments, as well as the coreferential relationships between arguments across different events. This means it can recognize when “a man in a red plaid shirt” in one event is the same “man in a red plaid shirt” in a later event, even if their visual appearance changes due to camera angles or other factors.
The model’s architecture incorporates a node-graph hierarchical attention mechanism. This allows EventFormer to capture attention at two levels: the node level (interactions within a single event graph) and the graph level (interactions between different event graphs in a sequence). Instead of using simple linear layers, EventFormer employs graph neural networks (GNNs) to generate key, query, and value representations, enabling it to build more comprehensive event representations by understanding both internal argument interactions and relationships between them.
A key innovation in EventFormer is its **Coreference Encoding** mechanism. In videos, the same object or person might look different across various events, making it hard for models to identify them as the same entity. Coreference encoding helps the model capture these relationships by adding specific encodings to node embeddings, enhancing its ability to distinguish coreferential nodes despite visual variations.
Also Read:
- Gaze-VLM: Enhancing AI’s Understanding of Human Actions Through Eye Gaze
- Video-As-Prompt: A Unified Framework for Semantic Video Generation
Training and Performance
EventFormer is trained using a two-stage strategy. First, it undergoes pre-training with a random masking approach, where certain event graphs are masked, and the model learns to reconstruct them. This helps the model better understand event relationships and improves its reasoning abilities. Following this, a post-training phase fine-tunes the model specifically for the AVEP task, focusing on predicting the last event in a sequence.
Experiments demonstrate that EventFormer significantly outperforms several state-of-the-art video prediction models and even large vision-language models (LVLMs) on the AVEP task. For instance, it shows notable improvements in verb prediction accuracy and F1-scores for noun argument prediction. The ablation studies further confirm the effectiveness of each proposed component, including the GNN architectures, coreference encoding, and the two-stage training strategy.
This research marks a significant step forward in video understanding, providing both a challenging new task and a powerful model to address it. The AVEP dataset and EventFormer model establish a strong baseline for future research in predicting complex, action-centric events in videos. You can find more details about this work in the research paper.
