TLDR: MediQAl is a new French medical question answering dataset with over 32,000 questions from medical exams, designed to evaluate language models’ ability in factual recall and complex reasoning across 41 medical subjects. It includes multiple-choice and open-ended questions, addressing a critical need for non-English medical AI benchmarks and revealing a significant performance gap between understanding and reasoning tasks in current models.
The field of artificial intelligence in medicine is rapidly advancing, with large language models (LLMs) showing immense potential. However, a significant challenge has been the lack of diverse, high-quality datasets, especially for languages other than English. This gap is now being addressed with the introduction of MediQAl, a groundbreaking French medical question answering dataset.
What is MediQAl?
MediQAl is a comprehensive dataset specifically designed to evaluate how well language models can handle factual medical knowledge and complex reasoning in real-world clinical scenarios within a French context. Developed by Adrien Bazoge and his team, this dataset is a crucial step towards more inclusive and accurate medical AI.
Why is MediQAl Important?
Most existing medical AI benchmarks are heavily focused on English, which limits their applicability in multilingual healthcare settings. Medical practices, educational systems, and even legal standards can vary significantly by country. Therefore, a direct translation of English questions might not accurately reflect the challenges faced in other regions. MediQAl fills this void by providing a benchmark rooted in French medical examinations, reflecting the unique cultural and educational nuances of French medicine.
Dataset at a Glance
The MediQAl dataset boasts an impressive 32,603 questions, all sourced from official French medical licensing examinations. These questions span across 41 diverse medical subjects, from cardiology to genetics and pediatrics. To provide a thorough evaluation, the dataset includes three distinct question formats: Multiple-Choice Question with Unique Answer (MCQU), Multiple-Choice Question with Multiple Answers (MCQM), and Open-Ended Question with Short-Answer (OEQ).
A unique feature of MediQAl is that each question is categorized as either “Understanding” or “Reasoning.” This allows researchers to analyze models’ capabilities in simple factual recall versus more complex cognitive tasks that require multi-step reasoning.
How Was MediQAl Built?
The data for MediQAl was meticulously collected from publicly available websites and forums used by French medical students and professors for national medical examination (ECN) preparation. These exams are critical for medical students in France, assessing their knowledge and clinical reasoning. The questions, including clinical scenarios and answers, are manually created and verified by academic and hospital faculty members. The team behind MediQAl also implemented rigorous filtering and preprocessing steps to ensure data quality and relevance, and used advanced AI models like GPT-4o to categorize questions for understanding and reasoning.
Key Findings from the Evaluation
The researchers conducted an extensive evaluation of 14 large language models on MediQAl, including both proprietary and open-source models, as well as those specifically designed for reasoning. The evaluation revealed several important insights:
A consistent performance gap exists between questions requiring multi-step reasoning and those assessing factual recall. Models generally perform better on “Understanding” questions than on “Reasoning” questions.
Reasoning-based models, such as o3 and DeepSeek-R1, generally showed better performance than their “vanilla” counterparts, especially on reasoning tasks. This suggests that techniques aimed at enhancing reasoning capabilities are beneficial.
Evaluating open-ended questions is complex. The study used a combination of lexical metrics and an “LLM-as-Judge” approach, where another LLM (Gemini-2.0-Flash) assessed the quality of generated answers against expert references, providing a more nuanced evaluation.
Performance varied significantly across different medical subjects, indicating that LLMs might have varying strengths and weaknesses depending on the specific medical domain.
Also Read:
- MedIQA: A New Foundation Model for Medical Image Quality Assessment
- HypKG: Integrating Patient Data with Medical Knowledge Graphs for Enhanced Healthcare Predictions
Looking Ahead
MediQAl represents a significant contribution to the medical AI landscape, particularly for non-English languages. It provides a robust benchmark for assessing the capabilities of language models in French medical question answering. While state-of-the-art LLMs show promising results, the observed performance gap in reasoning tasks highlights that these models still require further development and human oversight for real-world clinical applications. This dataset is openly available on HuggingFace, encouraging further research and development in this critical area. You can find more details about the dataset and its evaluation in the full research paper available at this link.
