TLDR: A new survey provides the first comprehensive classification of how large AI models, known as foundation models, are being applied to Electroencephalography (EEG) analysis. It details their use across various domains, including decoding brain states, translating EEG into text, vision, and audio, and integrating multiple data types for enhanced understanding. The paper also identifies key challenges such as ensuring model generalization across individuals and verifying the interpretability of AI-generated outputs from brain signals, while proposing future research avenues to overcome these hurdles.
Electroencephalography (EEG) is a non-invasive method used to record the brain’s electrical activity. It captures multi-channel voltage signals from the scalp, which are often complex due to low signal-to-noise ratios and variations between individuals. Traditionally, analyzing these signals for tasks like identifying user intentions or understanding cognitive states has relied on manually designed features or deep learning models. However, these methods often require large amounts of high-quality labeled data, which is scarce in EEG research.
In recent years, a new approach has emerged with the rise of foundation models. These powerful neural networks, initially trained on vast datasets of text, images, or audio, are now being adapted for EEG analysis. They offer strong representational capabilities and the ability to generalize across different data types. This shift is transforming how we approach EEG analysis, moving beyond traditional signal processing to more advanced, multimodal integration and high-level cognitive inference.
However, the rapid adoption of these techniques has led to a somewhat disorganized research landscape, with various model roles and architectures. To address this, a new survey provides the first comprehensive, modality-oriented classification of foundation models in EEG analysis. This study systematically organizes research based on the output modalities of EEG decoding, including native EEG decoding, EEG-to-text, EEG-to-vision, EEG-to-audio, and broader multimodal frameworks. The researchers rigorously analyze the ideas, theoretical foundations, and architectural innovations within each category, while also highlighting challenges such as model interpretability and real-world applicability.
Unimodal EEG Decoding
This category focuses on tasks that use only EEG signals to understand a user’s internal cognitive states, intentions, or task labels. It’s the most established form of brain signal analysis and is applied in areas like brain-computer interfaces (BCI), neurorehabilitation, and cognitive monitoring. Examples include recognizing intentions (like motor imagery), decoding cognitive states (such as attention levels or emotions), and monitoring physiological and neurological conditions (like sleep stages or Parkinson’s disease). In these applications, foundation models act as advanced feature extractors or classifiers, providing more accurate insights than traditional methods.
EEG-to-Text Research
This area explores how brain signals can be translated into natural language. It allows for the generation, matching, or recognition of textual content directly from EEG. The field is divided into three main directions: Text Alignment and Semantic Retrieval, EEG-based Text Generation, and EEG-based Domain-Specific Text Understanding. Researchers use contrastive learning to align EEG signals with text in a shared semantic space, enabling tasks like open-vocabulary matching. For text generation, EEG encoders are combined with pre-trained language models like GPT to convert neural signals into fluent language. This has potential for free-form communication and cognitive decoding. Domain-specific understanding, particularly in fields like medicine, uses EEG to augment professional language comprehension.
EEG-to-Vision Research
This research aims to uncover the brain’s patterns related to visual content and use them to reconstruct, recognize, or retrieve images a user is seeing or imagining. Challenges like low spatial resolution and individual variability in EEG are addressed by leveraging visual models like CLIP to create structured semantic spaces. This improves image retrieval and classification. In image reconstruction, diffusion models are now dominant, guiding generative models to recreate visual content from encoded EEG signals. There’s also progress in generating videos and 3D images from EEG, extending decoding from static frames to dynamic, spatiotemporally rich representations.
EEG-to-Audio Research
This field focuses on identifying or reconstructing auditory information from EEG, including speech and music. While traditional high-fidelity audio decoding often relied on invasive recordings, non-invasive EEG is showing increasing promise. Applications include automatic speech recognition, auditory attention detection, and music-related affective modeling. EEG-based audio generation explores creating sounds or melodies directly from brain rhythms or cognitive states, functioning as a brain-music creative interface. Audio reconstruction from EEG involves building models to map EEG signals to acoustic representations, allowing for the recovery of perceived or imagined music and speech.
Also Read:
- Unlocking Reader Confusion: A Multimodal Approach Using Brain Waves and Eye Movements
- Advancing Parkinson’s Diagnosis with a Generalizable EEG Model
Multimodal EEG Analysis Tasks
This section highlights the integration of EEG with two or more other modalities, such as fMRI, text, audio, or physiological signals, into a unified framework. This approach enhances contextual understanding and semantic representation by leveraging diverse information sources. Tasks include Multimodal Perception (where additional signals enhance understanding), Multimodal Output (generating multiple output modalities simultaneously from EEG), Cross-Modal Representation (creating a unified semantic space for various neural signals), and Assisted Enhancement (where EEG acts as an auxiliary modality to improve perception or decision-making in systems primarily focused on other modalities). These integrations aim to improve model generalization and enable joint inference and multi-task learning.
The survey concludes by outlining significant challenges. These include insufficient cross-subject generalization due to individual variability, issues with the authenticity and interpretability of cross-modal alignment mechanisms, and a lack of research in EEG generative and inverse modeling (generating EEG from other inputs). Future directions suggest integrating multi-scale time-frequency analysis and brain-region priors into EEG encoders, developing biologically constrained attention layers for cross-modal alignment, and creating unified cross-task and cross-modal benchmarks for evaluation. The ultimate vision includes the development of EEG Digital Twins and generative EEG models to advance precise cognitive modeling and adaptive Brain-Computer Interfaces. For more detailed information, you can read the full research paper here.
