TLDR: MLlm-DR is a novel AI model that uses multimodal large language models to diagnose depression from interview videos. Unlike previous methods, it provides clear explanations for its diagnostic scores, making it more trustworthy for clinicians. It integrates a smaller language model for generating scores and rationales, and a module called LQ-former to process speech and visual cues. The model achieves state-of-the-art results on benchmark datasets, demonstrating its effectiveness in providing both accurate and explainable depression assessments.
Diagnosing depression accurately and efficiently is a critical challenge in mental healthcare. Traditionally, clinicians rely on structured interviews, which, while effective, can be time-consuming and influenced by subjective judgment. Automated systems have emerged to assist, analyzing multimodal information from interview videos to predict depression scores. However, a significant hurdle for these systems has been their lack of explainability – they often provide a score without detailing how it was determined, leading to skepticism among medical professionals and limiting their adoption in clinical practice.
The advent of Large Language Models (LLMs) has opened new possibilities for explainable diagnostics. LLMs excel at understanding complex dialogue and can generate detailed rationales. However, existing multimodal LLMs are not specifically trained on interview data, which is crucial for accurate depression diagnosis, as it involves not just conversational content but also objective cues like voice and visual expressions.
Introducing MLlm-DR: A Novel Approach to Explainable Depression Recognition
A new research paper introduces MLlm-DR, a novel multimodal large language model designed to address these limitations. MLlm-DR aims to understand diverse information inputs and provide explainable depression diagnoses. This model integrates a smaller LLM with a lightweight query module (LQ-former) to achieve its goals.
The core idea behind MLlm-DR is twofold: first, to enable the LLM to generate not just depression scores but also clear, logical explanations for those scores. To achieve this, the smaller LLM is fine-tuned using a specially constructed training dataset. This dataset is created by leveraging advanced LLMs to generate evaluation rationales based on dialogue content, ensuring that the smaller model learns to provide coherent and reliable explanations.
Second, MLlm-DR enhances its ability to process multimodal information beyond just text. This is where the LQ-former comes into play. The LQ-former is designed to extract depression-related features from speech and visual data (like facial expressions and body language). It then maps these features into a unified text feature space, making them compatible with the LLM. This allows the LLM to consider a comprehensive set of indicators for depression, mirroring how clinicians assess patients.
How MLlm-DR Works
The MLlm-DR operates in two main stages. In the first stage, the LQ-former is pre-trained to extract relevant features from audio and visual inputs. These extracted features are then fed into a frozen LLM to predict depression scores, effectively teaching the LQ-former to identify depression-related cues. In the second stage, the pre-trained LQ-former’s parameters are frozen, and its extracted features are combined with transcribed text. This combined input is then used to fine-tune the LLM using a joint optimization strategy. This strategy combines language modeling loss (for generating rationales) and regression loss (for predicting scores), ensuring both accurate predictions and meaningful explanations.
Key Contributions and Performance
The researchers highlight several key contributions of MLlm-DR:
- It is the first multimodal LLM specifically designed for explainable depression recognition.
- It uses a unique method to construct an explainable depression assessment training dataset.
- The LQ-former module effectively integrates multimodal information for a comprehensive diagnosis.
- The model achieves state-of-the-art results on two benchmark interview-based depression datasets: CMDC and E-DAIC-WOZ.
Experimental results demonstrate MLlm-DR’s superior performance compared to existing methods. On the CMDC dataset, it achieved exceptional precision, recall, and F1 scores, while also significantly outperforming other methods on the E-DAIC-WOZ dataset across various metrics. This success is largely attributed to the LLM’s powerful text understanding capabilities combined with the multimodal integration provided by the LQ-former.
Ablation studies confirmed the importance of both the LQ-former module and the multi-task learning strategy, showing that removing either component led to a decline in performance. Furthermore, human evaluations involving clinical experts showed high agreement between MLlm-DR’s predictions and expert assessments, with a significant proportion of the model’s reasoning being rated as “fully agree” by experts. This underscores the practical value and interpretability of the approach.
Also Read:
- HopeBot: An AI Chatbot for Interactive Depression Screening Shows Promising Results
- Revolutionizing Medical Diagnosis: How AI’s KERAP Framework Offers Accurate Zero-Shot Predictions
Future Directions
While MLlm-DR represents a significant step forward, the researchers acknowledge that the current training dataset is primarily text-based, which might lead to a loss of fine-grained emotional information from speech and visual cues. Future work will focus on creating larger-scale, fine-grained depression label sets for speech and visual data to further enhance the model’s capabilities. This research holds significant practical application value, aligning closely with clinical needs for more transparent and comprehensive depression diagnosis. You can read the full research paper here.
