TLDR: MedARC’s team developed a pipeline for the Algonauts 2025 challenge to predict brain responses (fMRI) to natural movies. Their solution combines features from various state-of-the-art multimodal AI models (video, speech, text, vision-text, vision-text-audio), projects them into a latent space, aligns them with fMRI data, and maps them to brain regions using a lightweight encoder. Through extensive model selection and ensembling, their approach achieved a high Pearson’s correlation score, demonstrating the effectiveness of combining diverse multimodal features and a simple architecture for generalizing brain encoding models to novel movie stimuli.
Understanding how our brains react to the rich, dynamic world of movies is a fascinating challenge. The Algonauts Challenge, a biennial competition, aims to push the boundaries of this understanding by encouraging researchers to develop the best models for predicting neural activations from stimuli. The 2025 challenge specifically focused on long, multimodal movies, incorporating synchronized video, audio, and language.
MedARC’s team presented a sophisticated solution to this challenge, securing fourth place with a mean Pearson’s correlation of 0.2085 on a test set of previously unseen movies. Their approach highlights the power of combining diverse information sources and smart model design.
The MedARC Approach: A Multimodal Symphony
The core of MedARC’s pipeline involved leveraging rich multimodal representations from several cutting-edge pre-trained models. Imagine these models as specialized experts, each focusing on a different aspect of the movie:
- V-JEPA2: A vision model that understands video dynamics.
- Whisper: An audio model adept at processing speech and general sounds.
- Llama 3.2: A language model that processes the movie’s transcript.
- InternVL3: A vision-text model that integrates visual and textual information.
- Qwen2.5-Omni: A comprehensive model that combines vision, text, and audio.
These models extracted high-dimensional features from the movies. These features were then linearly projected into a smaller, more manageable space, carefully aligned with the fMRI (functional magnetic resonance imaging) brain activity time series, and finally mapped to specific brain regions using a lightweight encoder. This encoder had a clever design: a shared ‘group head’ that captures common brain responses across subjects, plus ‘subject-specific residual heads’ that account for individual differences.
Key Innovations and Findings
The team identified three main contributions to their success:
- Multimodal Feature Extraction: By using a variety of pre-trained models, they captured a comprehensive understanding of the movie’s content across different senses.
- Parameter-Efficient Architecture: Their model design was efficient, allowing for effective temporal alignment of features with fMRI data and adapting to individual subjects while sharing common components.
- Comprehensive Model Selection and Ensembling: They trained hundreds of model variations and combined the best-performing ones for each brain region in each subject. This ‘ensemble’ approach significantly improved the model’s ability to generalize to new, unseen movie stimuli.
Ablation studies, where components of the model were systematically removed or altered, revealed important insights. For instance, temporally aligning features with fMRI using 1D convolutions was crucial for good performance. Training the model on data from multiple subjects together also significantly improved results compared to training on single subjects. Furthermore, while a shared brain response across people is predictable, adding subject-specific components to the model was essential for maximizing accuracy.
Interestingly, the multimodal models like InternVL3 and Qwen2.5-Omni, which integrate information from multiple senses, showed the best individual performance. Each feature set, however, contributed uniquely to the overall model’s accuracy, with the language model (Llama 3.2) showing a strong, unique contribution.
Also Read:
- Advancing Alzheimer’s Diagnosis with OmniBrain’s Multimodal AI
- NeuroCLIP: A New AI Framework for Objective Methamphetamine Addiction Analysis and Treatment Monitoring
Ensemble Power and Future Optimizations
The ensemble strategy proved to be a game-changer, providing a substantial increase in performance. The team’s official submission used a ‘Top-k=5’ ensemble, but they discovered that increasing the ensemble size to ‘Top-k=20’ would have pushed them to second place in the competition, underscoring the value of large-scale ensembling for generalization.
The researchers also noted a fascinating post-competition discovery: by analyzing movie-specific scores provided by the competition platform, they realized they could have selected different models optimized for particular movie types (e.g., a model without language features for a silent film like ‘Chaplin’). This ‘movie-specific’ selection could have further boosted their scores, highlighting a potential future direction for adaptive brain encoding models.
In conclusion, MedARC’s work demonstrates that combining features from models trained on different modalities, using a simple yet effective architecture that accounts for both shared and individual brain responses, and employing comprehensive model selection and ensembling, significantly improves the ability to predict brain activity in response to complex, natural movie stimuli. Their code is available for further exploration and development. You can find it here: MedARC-AI/algonauts2025.
