TLDR: Researchers have developed a multi-stage large language model (LLM) framework to accurately and transparently extract suicide-related social determinants of health (SDoH) from unstructured text. This framework, which includes context retrieval, relevance verification, and SDoH factor extraction, outperformed existing models in identifying both common and rare factors. A pilot study also showed that providing AI-assisted explanations helped human annotators work faster and more accurately, demonstrating the potential for improved early intervention and prevention strategies.
Understanding the complex factors that contribute to suicide is crucial for developing effective prevention strategies. Much of this vital information, known as Social Determinants of Health (SDoH), is often buried within unstructured text narratives, such as clinical notes, making it difficult to access and analyze. Traditional data-driven approaches face significant hurdles, including the challenge of identifying rare but critical factors, capturing the temporal context of stressors leading up to an incident, and the inherent ‘black-box’ nature of many advanced AI models that makes their reasoning hard to interpret.
A New Approach to Health Data Extraction
To address these challenges, a team of researchers has developed an innovative multi-stage large language model (LLM) framework. This framework is designed to enhance the extraction of suicide-related SDoH factors from unstructured text in a scalable and generalizable manner. The study specifically focuses on SDoH factors that occur within two weeks prior to suicide incidents, aiming to identify critical windows for intervention.
The framework operates in three distinct stages:
- Context Retrieval: This initial stage uses an LLM to identify and extract sentences from the input text that are relevant to a specific SDoH factor. This helps narrow down the focus to pertinent information.
- Relevance Verification: Even with careful prompting, LLMs can sometimes select irrelevant sentences. This stage introduces an additional verification step, using another LLM (or a fine-tuned smaller model) to re-assess the relevance of the retrieved context, ensuring a high-quality set of sentences for the next step.
- SDoH Factor Extraction: In the final stage, an LLM processes the verified relevant sentences to extract the specific SDoH factors that occurred within the critical two-week period before the suicide incident.
Evaluating Performance and Transparency
The researchers rigorously evaluated their framework using data from the National Violent Death Reporting System (NVDRS), a comprehensive dataset of suicide death incidents in the U.S. and its territories from 2003 to 2019. They focused on extracting 10 infrequent and 8 frequent suicide-related SDoH factors.
The results showed that the proposed multi-stage framework consistently outperformed several state-of-the-art baseline models, including a fine-tuned BioBERT model and different configurations of GPT-3.5-turbo. Notably, it demonstrated significant improvements in extracting infrequent SDoH factors, which are often missed by conventional models due to their rarity. While compared to reasoning models like DeepSeek-R1, the framework showed higher recall, indicating its effectiveness in covering more relevant information, even if precision might vary depending on the factor.
A key advantage of this multi-stage design is its enhanced explainability. By breaking down the extraction process into intermediate steps, the framework provides explicit, human-interpretable outputs at each stage. For instance, the relevance verification step alone improved the accuracy of relevant context retrieval by 13.8%, making the model’s reasoning more transparent and trustworthy.
Real-World Impact: A Pilot User Study
To assess the practical value of their system, the researchers conducted a pilot user study with human experts tasked with annotating SDoH factors. The study compared annotation performance with and without AI assistance (where the framework highlighted relevant context). The findings revealed that experts using the AI-assisted method completed annotations faster, saving an average of 62.39 seconds per incident, and achieved slightly higher accuracy. Participants also reported lower mental stress and higher satisfaction when using the AI assistance, expressing a preference for the AI-assisted method for decision-making support.
Also Read:
- From Data Overload to Knowledge Clarity: A Quadruple Shot Approach to Medical Information
- New AI Method Improves Depression Screening via Drawing Projection Tests
Future Directions and Limitations
While promising, the study acknowledges several limitations. The fine-tuned smaller models used in some stages do not inherently provide reasoning, and the evaluation covered only a subset of the over 600 SDoH factors in the NVDRS. Future research will need to explore broader factor inclusion, assess fairness across diverse demographic groups, and further refine the concept of explainability in AI. Despite these, this multi-stage LLM framework represents a significant step forward in leveraging AI for public health, offering a more accurate and transparent way to identify individuals at risk and inform more effective suicide prevention strategies. For more detailed information, you can refer to the full research paper: A Multi-Stage Large Language Model Framework for Extracting Suicide-Related Social Determinants of Health.
