TLDR: VIBE is a two-stage Transformer model that predicts fMRI brain activity by fusing multi-modal video, audio, and text features. It achieved top results in the Algonauts 2025 Challenge, significantly outperforming baselines. The model’s success stems from its ability to integrate diverse sensory inputs, use temporal decoding with rotary embeddings, and employ techniques like ensembling and network-specific modeling, demonstrating strong generalization across various cinematic styles and languages.
Researchers have developed a groundbreaking model called VIBE (Video-Input Brain Encoder) that can predict human brain activity, specifically fMRI responses, by analyzing multi-modal inputs from videos, including visual, audio, and text features. This innovative two-stage Transformer architecture represents a significant step forward in understanding how our brains process complex, real-world sensory information.
The VIBE model was put to the test in the challenging Algonauts 2025 Challenge, where it achieved remarkable success. It secured first place in Phase 1 and an impressive second place overall in Phase 2, demonstrating its superior capability in modeling brain activity compared to existing benchmarks. The model was trained on 65 hours of movie data from the CNeuroMod dataset, which includes popular TV series like Friends and various films.
At its core, VIBE is designed to integrate diverse sensory information. It utilizes features extracted from several open-source models. For text, it leverages the powerful Qwen2.5 14B model, which processes entire video transcripts to capture long-term narrative and emotional contexts. Visual features are derived from models like V-JEPA 2, which focuses on action prediction, and SlowFast R101, which captures both spatial semantics and motion. Audio information is extracted using BEATs for general acoustic representations and Whisper-V3 for phonetically sensitive features. Additionally, Qwen2.5 Omni 3B provides multi-modal ‘omni’ features, and LaBSE is used for language-agnostic text features, particularly for out-of-distribution content like French-language clips.
The VIBE architecture consists of two main components: a Modality Fusion Transformer and a Prediction Transformer. The Modality Fusion Transformer is responsible for combining features from different sensory modalities at each specific moment in time, ensuring that visual, audio, and text information are integrated. This fused information is then passed to the Prediction Transformer, which models the temporal dependencies across time points, similar to how large language models process sequences. A key innovation in this stage is the use of Rotary Positional Embeddings (RoPE), which significantly improved the model’s ability to understand the relative timing of events.
Several strategies contributed to VIBE’s high predictive performance. Ensembling, which involves averaging the outputs of 20 independently trained models, proved to be the most effective method for boosting accuracy. The researchers also found benefits in training specialized models for specific functional brain networks, such as the visual cortex and the default mode network, as these areas exhibit distinct processing patterns. Interestingly, allowing the model to ‘look ahead’ in time by removing the causal mask from the prediction transformer also slightly improved performance, aligning with theories of predictive coding in the brain.
VIBE achieved a mean parcel-wise Pearson correlation of 0.3225 on in-distribution data (Friends S07) and 0.2125 on six out-of-distribution films. This significantly outperforms the official baseline model, which scored 0.20328 and 0.08952 respectively. The model’s ability to generalize to diverse content, including animated shorts, stick-figure drawings, black-and-white footage, and French-language clips, highlights its capacity to capture modality-specific yet content-agnostic representations.
Further analysis using Multiperturbation Shapley-Value Analysis revealed which features were most predictive for different brain regions. Text features from Qwen 2.5 14B were most influential in higher-order language and default-mode areas, while BEATs and Whisper features strongly correlated with auditory cortex. Visual encoders like SlowFast and V-JEPA highlighted occipital and motion-sensitive regions. This mapping provides valuable insights into the functional roles of different brain areas.
Also Read:
- Decoding Brain Activity: A New Approach to Visual Reconstruction from fMRI
- Unlocking Video Anomaly Detection with MLLMs’ Hidden Insights
In conclusion, VIBE offers a state-of-the-art solution for modeling fMRI responses to naturalistic video stimuli. Its robust architecture and effective integration of multi-modal features pave the way for a deeper understanding of human brain function under complex, real-world conditions. For more detailed information, you can refer to the research paper.
