TLDR: DentalBench is the first comprehensive bilingual benchmark and corpus designed to evaluate and improve large language models (LLMs) in the specialized field of dentistry. It comprises DentalQA, a question-answering benchmark with 36,597 questions across 16 dental subfields, and DentalCorpus, a large dataset for domain adaptation. Evaluations of 14 LLMs revealed significant performance gaps, particularly in open-ended tasks, but demonstrated that domain adaptation using DentalCorpus substantially enhances LLM capabilities in dentistry.
Recent advancements in large language models, often referred to as LLMs, have shown remarkable capabilities across various domains, including general medicine. However, their performance in highly specialized medical fields like dentistry has remained largely unexplored. This gap is primarily due to a lack of dedicated evaluation resources tailored to the unique and deep domain-specific knowledge required in oral healthcare.
To address this critical need, researchers have introduced DentalBench, the first comprehensive bilingual benchmark designed specifically to evaluate and advance LLMs in the dental domain. This innovative benchmark aims to provide a robust framework for understanding the current limitations of AI in dentistry and guiding the development of more reliable systems for clinical applications.
DentalBench is composed of two main elements. The first is DentalQA, an extensive English-Chinese question-answering benchmark. It features 36,597 questions that cover four distinct task formats and span 16 specialized dental subfields, ranging from oral anatomy to periodontics and orthodontics. These task formats include single-answer multiple-choice questions (MCQ), multi-answer multiple-choice questions (MAQ, currently Chinese only), open-ended questions (OEQ) simulating clinical scenarios, and terminology definition questions (DEF).
The second component is DentalCorpus, a substantial and high-quality bilingual corpus. Curated specifically for dental domain adaptation, it contains 337.35 million tokens. This corpus is instrumental for both supervised fine-tuning (SFT) and retrieval-augmented generation (RAG), methods crucial for enhancing an LLM’s understanding and generation capabilities within a specific field.
The creators of DentalBench evaluated 14 different LLMs, including proprietary models, open-source models, and those specifically designed for medical applications. Their findings revealed significant performance disparities among these models, varying considerably based on the task type and the language used. Notably, open-ended tasks like OEQ and DEF proved to be much more challenging for current models compared to multiple-choice questions, highlighting a persistent difficulty in domain-specific generative reasoning and terminology understanding.
Interestingly, the study observed that general-purpose LLMs often outperformed medical-specific LLMs on multiple-choice tasks related to dentistry. This suggests that existing medical tuning might not adequately capture the nuanced, factual knowledge specific to dentistry. Furthermore, while increasing model scale (as seen within the Qwen-2.5 family) improved performance on multiple-choice questions, it offered limited gains for open-ended tasks, indicating that factual recall benefits more from model size than complex generative reasoning.
A key finding from the research was the substantial positive impact of domain adaptation using DentalCorpus. Both supervised fine-tuning and retrieval-augmented generation strategies significantly improved model performance. RAG, in particular, showed a greater impact on open-ended tasks, especially in Chinese. Combining both SFT and RAG yielded additive benefits, particularly for multiple-choice questions. These results underscore the critical importance of domain-specific benchmarks and data for developing effective and trustworthy LLMs tailored for specialized healthcare applications.
Also Read:
- Unpacking Privacy in AI: A New Benchmark for Domain-Specific Synthetic Text
- Mapping Medical Knowledge: A Graph-Based System for Evaluating AI in Healthcare
In conclusion, DentalBench serves as a vital resource for evaluating knowledge-grounded language models in dentistry, fostering a deeper understanding of oral healthcare, and encouraging further research in this specialized field. For more detailed information, you can refer to the full research paper here.
