TLDR: Doctor Sun is a new bilingual (Chinese-English) multimodal AI model designed for biomedical tasks. It integrates text and image data using a specialized medical large language model and a vision encoder, trained in two stages for feature alignment and instruction tuning. The model demonstrates strong performance in medical question answering, visual question answering, and image classification, often outperforming existing models. It also introduces SunMed-VL, a new bilingual medical dataset. While promising for diagnostic assistance, it’s not yet ready for direct clinical use.
The field of artificial intelligence is making significant strides in healthcare, particularly with the emergence of large multimodal models (LMMs). These models hold immense promise for various biomedical tasks, from analyzing pathology to generating radiology reports and assisting in general biomedical applications. However, existing AI solutions in this area often face challenges, such as a limited understanding of complex medical concepts due to reliance on general foundation models and difficulties in effectively linking text with images.
To address these limitations, researchers have introduced Doctor Sun, a specialized large multimodal generative model designed specifically for medicine. Doctor Sun is engineered to process, integrate, and interpret diverse biomedical data, including both text and images. It achieves this by combining a pre-trained vision encoder with a medical-specific large language model (LLM). The model undergoes a unique two-stage training process that focuses on aligning features between different data types and fine-tuning instructions.
A significant contribution of this research is the release of SunMed-VL, a comprehensive bilingual medical multimodal dataset. This dataset, along with all associated models, code, and resources, is made freely available to support and accelerate advancements in biomedical multimodal research.
Traditional large language models primarily rely on text, which can be a significant limitation in medical diagnostics where visual information, such as radiological scans or pathological slides, is crucial. Doctor Sun aims to overcome this by integrating multimodal data, allowing for a more complete understanding of disease manifestations. This integration helps capture a broader spectrum of conditions, leading to more nuanced insights.
Previous medical multimodal LLMs often had limitations, such as being restricted to a single language or focusing on specific illnesses. For instance, some models were designed mainly for lung disease diagnosis and were only available in English. Doctor Sun, however, is a bilingual (Chinese-English) model, making it more versatile for real-world medical practice that often involves diverse and complex disease patterns across multiple specialties like geriatric medicine, internal medicine, and palliative care.
The architecture of Doctor Sun involves three main components: a text foundation model for logical reasoning and clinical decision-making, a visual foundation model to extract features and identify abnormalities in medical images, and a cross-modal projector that aligns visual data with textual semantic space. This design allows for seamless integration of imaging findings with clinical notes, providing a holistic view of patient conditions. The model is trained on a high-quality bilingual dataset from public sources, including radiology images, pathology slides, and clinical photographs with corresponding Chinese and English annotations.
To ensure its expertise, the general-purpose language foundation model within Doctor Sun is first optimized to accumulate fundamental medical knowledge. Following this, the entire model undergoes a two-stage training strategy: feature alignment and instruction tuning. This approach ensures proficiency in multimodal medical diagnostic tasks while retaining general-purpose capabilities.
The researchers emphasize that Doctor Sun achieves a nuanced understanding of pathological images and texts, enabling complex pathological diagnoses. Empirical studies show its superior performance across ten datasets spanning three tasks, demonstrating excellent zero-shot evaluation results and surpassing state-of-the-art benchmarks. The study also investigated the optimal ratio of domain-specific to generic data during training, finding a balance that enhances medical diagnostic proficiency while maintaining general capabilities.
Doctor Sun’s performance was rigorously evaluated against other leading medical multimodal models like RadFM and LLaVA-Med. In question answering tasks, Doctor Sun showed significant improvements, indicating a deeper integration of medical expertise. For visual question answering, it demonstrated a substantial advantage, especially on benchmarks like Path-VQA, achieving superior performance with fewer parameters. In image classification tasks, Doctor Sun also showed comparable or slightly better performance.
The research highlights the importance of integrating domain-specific knowledge into both the visual encoder and the language model components of multimodal AI. It also underscores the necessity of combining general data with domain-specific data to balance specialized and general abilities, which is crucial for real-world clinical applications. The bilingual capability further enhances Doctor Sun’s utility in multilingual medical environments.
Also Read:
- MultiMedEdit: A New Benchmark for Updating Medical AI Knowledge
- GPT-5 Achieves Breakthrough in Multimodal Medical Reasoning, Outperforming Human Experts
While Doctor Sun shows promising results, the researchers acknowledge that it has not yet been evaluated in a real-world clinical environment and is not suitable for direct clinical application at this stage. Future work will involve more extensive testing to explore the impact of integrating domain-specific and generic data in other specialized domains. This pioneering research marks a significant step towards more reliable and accurate medical diagnosis through advanced AI. For more details, you can refer to the full research paper: Doctor Sun Research Paper.
