TLDR: MedFact is a new benchmark for evaluating Large Language Models (LLMs) on Chinese medical fact-checking. It uses 2,116 expert-annotated texts across diverse medical specialties and error types. The study found that LLMs struggle with precisely locating factual errors and exhibit an “over-criticism” tendency, where they misidentify correct information as erroneous. Retrieval-Augmented Generation (RAG) improved performance, but a significant gap remains between LLM and human expert capabilities, highlighting the need for more medically reliable AI.
Large Language Models, or LLMs, are becoming more common in healthcare. This means it’s really important to know if they can be trusted to provide accurate medical information. A new study introduces MedFact, a benchmark designed specifically to test how well LLMs can fact-check Chinese medical texts.
The researchers, including Jiayi He and Yangmin Huang, created MedFact to fill a gap in existing evaluations. Previous benchmarks often used limited data or synthetic texts, which didn’t truly reflect the complexity of real-world medical information. MedFact, on the other hand, uses 2,116 instances of medical texts that were carefully reviewed and annotated by medical experts. These texts cover 13 different medical specialties, 8 types of errors, 4 writing styles, and various difficulty levels, making it a comprehensive and challenging dataset.
Building MedFact involved a unique process that combined artificial intelligence with human expertise. An AI system initially filtered a large corpus of medical texts, and then medical professionals provided iterative feedback to refine this filtering. This human-in-the-loop approach ensured both high quality and difficulty for the dataset. The data was sourced from medical encyclopedias and consultation platforms, and importantly, it’s proprietary, meaning it’s unlikely to have been used in the pre-training of many LLMs, ensuring a fair evaluation.
The study evaluated 20 leading LLMs, both open-source and proprietary, on two main tasks: veracity classification (determining if a text is correct or incorrect) and error localization (pinpointing the exact error). The results showed that while LLMs can often tell if a text has an error, they struggle significantly with identifying the precise location of that error. Even the best-performing models fell short of human expert performance. For instance, the F1 score for error localization for the top model was 0.6858, still below the human baseline of 0.7012.
An interesting finding was the “over-criticism” phenomenon. Advanced reasoning techniques, such as multi-agent collaboration, sometimes caused models to incorrectly flag accurate information as erroneous. This tendency to be overly skeptical can undermine reliability, especially in critical medical applications where distinguishing truth from falsehood is paramount.
However, some strategies did help. Retrieval-Augmented Generation (RAG), which involves providing models with external, domain-specific knowledge, significantly improved performance on both tasks. This suggests that giving LLMs access to reliable external information is a very effective way to enhance their medical fact-checking abilities.
The analysis also revealed that a lack of deep medical knowledge was a primary reason for model mistakes, accounting for a large percentage of errors. Models often struggled with nuanced, specialized medical understanding, conflating concepts or relying on outdated information. The study also found that models were better at detecting errors in fabricated misinformation than in professionally curated encyclopedia articles, highlighting their difficulty with subtle inaccuracies.
Also Read:
- AI’s Role in Clinical Trial Recruitment: Insights from Social Media Analysis
- Improving RAG Performance with Hierarchical Document Chunking
In conclusion, MedFact provides a robust tool for assessing LLMs in Chinese medical fact-checking. The findings underscore that while LLMs show promise, there’s still considerable work needed to develop models that are truly factually reliable and medically aware enough for safe deployment in healthcare. You can read the full research paper for more details here.
