TLDR: A new brain encoding model uses a network-specific approach, dividing the brain into functional clusters and training separate AI models for each to predict responses to multimodal movie stimuli. This method significantly improved prediction accuracy, especially for out-of-distribution content, by adapting to the unique temporal and sensory processing of different brain networks.
A central goal in understanding the human brain is to model how it responds to the rich, complex stimuli we encounter in the real world, such as watching a movie. Traditionally, brain models have focused on individual senses, but real-world perception involves integrating visual, auditory, and linguistic information simultaneously across various brain networks.
Researchers have introduced a novel approach to predict brain responses to complex multimodal movies. Instead of treating the brain as a single, uniform system, their method acknowledges its intricate functional organization. They leveraged a well-known brain map, the Yeo 7-network parcellation of the Schaefer atlas, which divides the brain into seven large-scale functional networks.
The core of their strategy involved grouping these seven functional networks into four distinct clusters. For each of these clusters, they trained separate multi-subject, multi-layer perceptron (MLP) models. This innovative architecture allows for specialized optimization for each cluster, meaning each model can adapt its temporal dynamics and how it weights different sensory modalities based on the specific functional role of its target network. For instance, some networks might be more sensitive to rapid visual changes, while others integrate information over longer timescales.
To handle data from multiple individuals, the model uses a shared ‘backbone’ network that learns common patterns across subjects, combined with subject-specific prediction heads that account for individual differences in brain organization. Furthermore, the researchers incorporated ‘memory modeling’ into their approach. This means the models consider information from past stimuli, not just the current moment, which is crucial because brain networks exhibit varying temporal receptive windows. This memory component was particularly beneficial for the Visual, Somatomotor, and Dorsal Attention networks.
The process began with extracting features from the movie stimuli. For visual information, they used models like ViNET (which focuses on salient, attention-grabbing areas) and VideoMAE2 (for high-level temporal and spatial dynamics). Audio features were captured using Wav2Vec2.0 (for speech), openSMILE (for low-level acoustic properties), and AudioPANNs (for non-speech sounds like environmental noises and music). Language features were extracted using RoBERTa-base, a powerful transformer model, focusing on contextualized word embeddings and attention weights.
The effectiveness of this clustered strategy was demonstrated through its performance in the Algonauts Project 2025 Challenge, a competition designed to assess computational models based on their ability to predict brain responses to multimodal movie content. The model achieved an impressive eighth-place ranking. Notably, its out-of-distribution (OOD) correlation scores—meaning its ability to predict responses to entirely new movie content not seen during training—were nearly double those of the baseline model.
Analysis of the results showed that the model was particularly accurate in predicting activity in auditory and language-processing areas, such as the superior temporal regions. While performance naturally declined when moving from in-distribution (familiar content) to out-of-distribution (unseen content), the spatial pattern of predictable regions remained consistent, with language and auditory areas maintaining higher accuracy compared to visual regions. This suggests that the feature extraction methods successfully captured generalizable representations for audio-linguistic processing.
Also Read:
- AI Learns from the Brain: Fine-Tuning LLMs for Mental Workload Detection
- Efficient AI Models Advance Brain MRI Analysis with Limited Data
This work represents a significant step forward in understanding how the brain processes complex, real-world stimuli. By embracing the brain’s functional organization and employing specialized models for different networks, the researchers have developed a powerful framework for predicting neural responses. The code for this research is publicly available for further exploration. You can find the full research paper here: Network-Specific Models for Multimodal Brain Response Prediction.
