TLDR: SpeechCARE is a new AI system that uses multimodal speech processing, combining acoustic and linguistic features from advanced transformer models with demographic data, to detect cognitive impairment like Mild Cognitive Impairment (MCI) and Alzheimer’s disease. It features a novel Adaptive Gating Fusion architecture for effective integration, robust preprocessing including LLM-assisted anomaly detection, and an explainability framework. SpeechCARE achieved high accuracy (AUC=0.90 for MCI detection) and addressed biases, showing promise for accessible, non-invasive early diagnosis in real-world healthcare settings.
Alzheimer’s disease and related dementias (ADRD) pose a significant public health challenge, affecting a large portion of adults over 60. A major concern is that more than half of individuals experiencing cognitive decline, including mild cognitive impairment (MCI), remain undiagnosed. Early detection is crucial for timely intervention, and recent research has highlighted the potential of speech-based assessments in this area.
Speech patterns can reveal subtle changes linked to cognitive impairment. For instance, phonetic motor planning deficits can affect vocal tract control, altering acoustic features like pitch and tone. Memory and language difficulties can lead to errors in language organization, reduced fluency, and syntactic or semantic mistakes. However, traditional speech processing methods often fall short, exhibiting limited performance and generalizability across different languages and speech contexts.
Introducing SpeechCARE: A Multimodal Approach
To address these limitations, researchers have developed SpeechCARE, a groundbreaking multimodal speech processing pipeline. This innovative system leverages advanced, pre-trained, multilingual acoustic and linguistic transformer models to capture the nuanced acoustic and linguistic cues associated with cognitive impairment. At its heart is a novel multimodal fusion architecture, inspired by the Mixture of Experts (MoE) paradigm, which dynamically weighs these acoustic and linguistic features for effective integration. This design not only enhances performance but also improves generalizability across various speech production tasks, such as story recall and sentence reading. A key advantage of SpeechCARE is its ability to seamlessly incorporate additional data, like social determinants of health or MRI scans, further boosting its sensitivity across the entire spectrum of cognitive impairment.
SpeechCARE is designed to overcome challenges posed by small sample sizes, allowing for the inclusion of diverse linguistic populations often overlooked in research. Its robust preprocessing pipeline includes automatic transcription using state-of-the-art models like Whisper-Large, and employs Large Language Models (LLMs) for tasks such as data anomaly detection and speech task identification. Furthermore, SpeechCARE features an explainability framework that visualizes each modality’s contribution to decision-making, highlighting specific linguistic and acoustic cues linked to cognitive impairment through a novel SHAP-based approach and LLM-based reasoning.
Performance and Fairness
The system has shown promising results. In distinguishing between cognitively healthy individuals, those with MCI, and those with AD, SpeechCARE achieved an Area Under the Curve (AUC) of 0.88 and an F1 score of 0.72. Specifically for detecting MCI against a control group, it reached an impressive AUC of 0.90 and an F1 score of 0.62. These metrics indicate a strong capability for early detection.
Recognizing the importance of fairness, SpeechCARE also underwent rigorous bias analyses. While no significant demographic biases were observed across most groups, a slight bias was noted for individuals over 80 years old. Dataset constraints also introduced biases for Mandarin speakers (all of whom had MCI in the dataset) and Spanish speakers (who only performed sentence reading tasks, limiting the capture of critical speech cues). To mitigate these issues, the team applied various techniques, including oversampling, frequency masking for speech augmentation, and replacing certain language models with more generalized multilingual alternatives. These efforts significantly improved fairness metrics, particularly for the age-over-80 group and Spanish speakers.
The Technology Behind SpeechCARE
The methodology involved a comprehensive evaluation of various speech processing models. The core components selected for SpeechCARE’s feature network were mGTE (a multilingual Generative Text Encoder) for linguistic analysis and mHuBERT (a multilingual variant of HuBERT) for acoustic analysis. These models were chosen for their extensive multilingual pre-training and high generalizability. The Adaptive Gating Fusion (AGF) network was identified as the most effective strategy for combining acoustic, linguistic, and demographic information, dynamically adjusting the weight of each modality based on its relevance. This dynamic adaptation, interpretability, efficiency, and robustness are key advantages of the AGF framework.
Also Read:
- Enhancing Language Models for Early Alzheimer’s Detection
- Enhancing Alzheimer’s Detection with Explicit Knowledge in Language Models
Looking Ahead
The future of SpeechCARE is focused on expanding its capabilities and real-world applicability. Researchers plan to integrate speech data with other biomarkers, electronic health record (EHR) data, and social determinants of health through collaborations with institutions like Columbia University’s Alzheimer’s Disease Research Center. There are also plans to fine-tune SpeechCARE on routine patient-clinician communications, enhancing its explainability for seamless integration into EHR systems and supporting clinician-centered design. For longitudinal monitoring of cognitive decline, a mobile application called “SpeechCARE Lite” is under development, which will allow for recording speech samples over time and integrating time-series models for analysis. Continuous improvements to noise reduction, transcription bias, and speaker diarization components are also in the pipeline.
SpeechCARE represents a significant step forward in the early detection of cognitive impairment, offering an accessible, non-invasive, and cost-effective solution for real-world care settings. For more detailed information, you can refer to the full research paper here.
