TLDR: This research introduces a novel brain decoding method that reconstructs visual stimuli from fMRI data by treating brain signals as 2D spherical data on the cortical surface. It integrates individual brain structure and uses a “positive-sample mixup” strategy to improve accuracy, interpretability, and cross-subject generalization, outperforming previous methods even with lower data resolution.
The human brain, often likened to a highly optimized neural network, processes sensory information into complex neural activity patterns. Brain decoding aims to reverse this process, reconstructing sensory stimuli from recorded brain activity, typically using functional magnetic resonance imaging (fMRI). This field has significant implications for understanding how the brain represents information and for developing brain-computer interfaces.
Historically, fMRI-image reconstruction methods have evolved from semantic category reconstruction to pixel-level reconstruction. A common approach involves training an fMRI encoder to generate embeddings that guide a diffusion model for image reconstruction. While successful, existing methods often simplify fMRI data as one-dimensional signals, overlooking crucial spatial and structural properties of the brain. fMRI signals originate from voxels on the cerebral cortex, naturally forming spatial patterns on a two-dimensional, non-Euclidean cortical surface. Additionally, individual brain structural differences significantly impact functional responses, leading to varied neural responses to identical stimuli, a challenge current methods struggle to capture for cross-subject generalization.
A new research paper, “From Flat to Round: Redefining Brain Decoding with Surface-Based fMRI and Cortex Structure,” introduces several innovations to address these limitations. The core of their approach is a novel “Sphere Tokenizer” that explicitly models fMRI signals as spatially coherent 2D spherical data on the cortical surface. This tokenizer processes fMRI signals into “fMRI tokens” that inherently incorporate spatial structural information. By mapping fMRI signals onto a standard sphere using FreeSurfer-based methods and employing spherical convolution, the model can extract fMRI features while considering the spatial organization of the brain.
A key contribution is the integration of structural MRI (sMRI) data, which allows for personalized encoding of individual anatomical variations. The Sphere Tokenizer incorporates cortical structure properties like thickness, surface area, sulcal depth, and curvature, enabling the model to perceive differences in brain structures across subjects. This is a significant step, as it’s the first time brain structure has been directly incorporated into the fMRI-image reconstruction task. Furthermore, a “sphere positional embedding” is introduced, based on 3D coordinates on the cortical surface, helping the tokenizer account for functional pattern differences across various cortical regions.
The paper also tackles a common issue in fMRI-image datasets: the asymmetry where one image corresponds to multiple fMRI scans. Previous methods often train on individual scans but infer using averaged scans, creating a data distribution gap. To mitigate this, the researchers propose a “positive-sample mixup strategy.” During training, multiple fMRI scans associated with the same visual stimulus are randomly mixed, mirroring the averaging process used during inference. This strategy efficiently leverages multiple fMRI scans, enhancing reconstruction accuracy and generalizability.
The overall framework involves a three-stage pipeline. First, the Sphere Tokenizer maps raw fMRI data to fMRI tokens, incorporating spatial and structural information. Second, an fMRI encoder takes these tokens and aligns them with CLIP (Contrastive Language–Image Pre-training) embeddings. Finally, an image decoder reconstructs the image from the aligned fMRI embedding. The fMRI encoder also utilizes a “token-wise alignment” approach, focusing loss calculation only on semantically meaningful parts of text embeddings, further refining the alignment process.
Experiments conducted on the Natural Scenes Dataset (NSD) demonstrate that this biologically informed approach achieves superior reconstruction performance compared to state-of-the-art methods, even when using lower fMRI resolution. The qualitative results show high similarity between reconstructed and ground truth fMRI data, indicating the Sphere Tokenizer’s ability to retain significant information. The model also exhibits strong cross-subject robustness, with similar fMRI heatmaps across different individuals, suggesting it learns subject-independent patterns.
Also Read:
- Advancing EEG-Based Emotion Recognition with Spatial-Temporal Transformers and Adaptive Learning
- Gaze-Guided Robots: Enhancing Efficiency and Robustness with Human-Inspired Vision
This research marks a significant advancement in brain decoding by moving beyond simplified 1D fMRI data representations. By explicitly modeling fMRI signals on the cortical surface and integrating individual brain structural information, the proposed methods enhance reconstruction accuracy, biological interpretability, and generalizability across individuals. For more details, you can refer to the full research paper available at this link.
