TLDR: Researchers have developed a novel method to analyze fMRI data by transforming complex 4D brain scans into 2D ‘flat maps.’ These flat maps are then fed into Vision Transformers using a masked autoencoder framework, which learns to reconstruct brain activity. The model, fm-MAE, demonstrates robust performance scaling with dataset size and achieves high accuracy in decoding cognitive states, sex classification, and visual decoding tasks, marking a significant step towards general-purpose fMRI foundation models.
A new study introduces an innovative approach to analyzing functional Magnetic Resonance Imaging (fMRI) data, paving the way for more effective brain activity analysis using modern deep learning techniques. Researchers have developed a method to transform complex 4D volumetric fMRI data into more manageable 2D “flat maps,” which are then processed by Vision Transformers (ViTs) within a spatiotemporal masked autoencoder (MAE) framework.
Functional MRI is a powerful tool that captures noisy 3D maps of brain activity over time. The challenge for deep learning models has been how to best represent this intricate data. Traditional methods often reduce the data’s dimensionality significantly, leading to potential information loss. Other approaches, while preserving full data, are computationally intensive and require models to learn the brain’s intrinsic structure from scratch.
The team, including Connor Lane, Daniel Z. Kaplan, Tanishq M. Abraham, and Paul S. Scotti, proposes an intermediate strategy: representing fMRI activity as a series of 2D maps on a flattened cortical surface. This “flat map” representation retains the full cortical fMRI signal while explicitly incorporating the natural organization of local cortical neighborhoods. Crucially, because these flat maps are essentially 2D images, they can be easily processed by standard Vision Transformers, which are highly effective in computer vision tasks.
To train these ViTs, the researchers adopted the spatiotemporal masked autoencoder (MAE) framework. In this setup, a large portion of the fMRI flat map “patches” are masked out, and the model is trained to reconstruct these missing parts based on the visible patches. This self-supervised learning approach allows the model to learn rich representations of brain activity without requiring extensive manual labeling.
The model, dubbed fm-MAE (flat map MAE), was pretrained on an extensive dataset of 2.3K hours of fMRI flat map videos from the Human Connectome Project (HCP). A significant finding was the observation of a strict power law scaling relationship between masked fMRI modeling performance and dataset size. This indicates that as more data is used for pretraining, the model’s ability to reconstruct brain activity patterns improves predictably, a hallmark of robust foundation models.
Beyond reconstruction, the fm-MAE model demonstrated promising results in various downstream decoding tasks. It showed strong performance in fine-grained cognitive state decoding across different subjects and in subject-specific trait decoding, such as sex classification. A new benchmark, NSD CLIP classification, which involves predicting global image labels from brain activity, also saw the fm-MAE model perform significantly better than chance and baseline models, even for unseen subjects and out-of-distribution data.
The researchers found that while increasing dataset size generally improved performance, increasing data diversity might be even more critical for learning truly generalizable representations. Interestingly, relatively smaller models were often sufficient, and reducing the temporal patch size (which increases the granularity of the model’s input) consistently improved performance across all benchmarks.
Also Read:
- New AI Tool TinyViT-Batten Improves Early Diagnosis of Batten Disease from Pediatric Brain Scans
- Advanced Machine Learning Framework Enhances Alzheimer’s Disease Progression Prediction
This work represents a significant step towards building foundation models for fMRI data, offering a high-fidelity yet structured representation that bridges the gap between complex brain data and powerful deep learning architectures. The project is part of an ongoing open science initiative, with code and datasets publicly available. For more technical details, you can refer to the full research paper here.
