TLDR: FocusMed is a new AI framework that uses large language models (LLMs) to improve the summarization of complex consumer health questions for doctors. It addresses common issues like identifying the main point of a question and preventing the AI from generating false information. The framework works by first extracting the core focus of a question, then fine-tuning LLMs with this enhanced information, and finally selecting the best summary based on faithfulness, conciseness, and coverage. FocusMed has achieved state-of-the-art results on medical question summarization datasets, making it easier for doctors to understand patient queries and provide accurate advice.
Online medical platforms have become a popular way for people to seek health advice. However, the questions asked by consumers often contain a lot of unnecessary information and non-medical terms, making it difficult for doctors to quickly understand the core issue and provide an accurate diagnosis. This challenge led to the development of Medical Question Summarization (MQS), a task aimed at transforming these lengthy consumer health questions (CHQs) into concise, doctor-friendly frequently asked questions (FAQs).
Existing methods for MQS have faced difficulties, particularly in accurately identifying the main focus of a question and preventing the AI model from generating incorrect or fabricated information, a phenomenon known as hallucination. To tackle these issues, researchers have introduced FocusMed, a new framework based on large language models (LLMs) designed to significantly improve the quality of medical question summaries.
How FocusMed Works
FocusMed employs a three-pronged approach to enhance medical question summarization:
1. Question Focus Extraction: The first step involves using LLMs to pinpoint the core focus of a consumer health question. The framework uses specially designed prompts to guide the LLMs in extracting key entities, primarily medications and symptoms, from the CHQs. To ensure the accuracy and reliability of this extraction, a faithfulness validation mechanism is in place. This mechanism checks if the extracted focus truly reflects the original text, helping to reduce hallucinations. The extracted core focuses are then combined with the original CHQ-FAQ pairs to create an enriched dataset, which is crucial for subsequent training.
2. Model Fine-tuning: With the enhanced dataset, popular LLMs like Qwen2.5-7B and LLaMA3.1-8B are fine-tuned. This process uses a technique called QLoRA, which efficiently optimizes these models to better understand and summarize medical questions. The goal is to enable the models to more accurately extract the key focus and minimize the generation of irrelevant or false information.
3. Multi-Dimensional Quality Evaluation and Selection: FocusMed doesn’t just rely on one summary. Instead, it generates multiple potential summaries using different combinations of models for extraction and fine-tuning. Each of these summaries is then rigorously evaluated across three critical dimensions: faithfulness (how factually consistent it is with the original question), conciseness (how efficiently it conveys essential information), and coverage (how well it captures all relevant information from the original text). The summary that achieves the highest overall weighted score across these dimensions is then selected as the final, optimal output.
Key Achievements and Impact
Experiments conducted on two widely-used MQS datasets, MEDIQA and MeqSum, demonstrated FocusMed’s superior performance. The framework achieved state-of-the-art results across all evaluation metrics, showing significant improvements in the model’s ability to identify the critical focus of questions and notably reducing instances of hallucination. For example, on the MEDIQA dataset, FocusMed showed a 7.3% improvement in ROUGE-L scores compared to previous state-of-the-art methods, and a 6.7% improvement on the MeqSum dataset.
This advancement means that healthcare professionals can more efficiently grasp the user’s intent from complex health questions, leading to more accurate and timely medical advice. For patients, it translates to clearer communication and potentially better care outcomes.
Also Read:
- Enhancing Access to UK Clinical Guidelines with AI: A RAG System for Healthcare
- DOCTOR-R1: An AI Agent for Empathetic and Strategic Clinical Conversations
Future Directions
While FocusMed represents a significant leap forward, the researchers acknowledge some limitations. The framework sometimes struggles with numerical details, such as specific durations or dosages, and can occasionally overemphasize minor details. Future work will focus on further optimizing the model’s handling of these specific details and enhancing its sensitivity to critical numerical information to ensure even greater accuracy in medical summaries.
For more in-depth information, you can read the full research paper: FocusMed: A Large Language Model-based Framework for Enhancing Medical Question Summarization with Focus Identification.
