TLDR: The BioASQ 2024 challenge, the twelfth edition, brought together 37 teams to advance large-scale biomedical semantic indexing and question answering. It featured four tasks: the established biomedical QA (Task 12b) and iterative QA for developing problems (Synergy 12), plus two new tasks—MultiCardioNER for multilingual clinical entity detection in cardiology, and BioNNE for nested named entity recognition in Russian and English. The challenge highlighted the strong performance of AI systems, particularly with Large Language Models and Retrieval Augmented Generation, and underscored the critical role of domain-specific data in improving results.
The BioASQ 2024 challenge, the twelfth in its series, recently concluded, marking another significant stride in the field of large-scale biomedical semantic indexing and question answering. Organized in the context of the Conference and Labs of the Evaluation Forum (CLEF) 2024, BioASQ continues its decade-long mission to push the boundaries of artificial intelligence in understanding and processing complex biomedical information. This year, the challenge engaged 37 competing teams, who collectively submitted over 700 distinct systems across four diverse shared tasks.
Four Core Challenges
BioASQ 2024 featured new editions of two established tasks and introduced two innovative ones:
-
Task 12b: Biomedical Question Answering This task focused on a comprehensive question-answering scenario, requiring participants to develop systems capable of handling all stages of biomedical QA. It addressed four question types: “yes/no,” “factoid,” “list,” and “summary” questions. The task was divided into three phases: Phase A for identifying relevant articles and snippets, Phase A+ for submitting exact and ideal answers in parallel with Phase A, and Phase B, where systems provided exact and ideal answers using manually selected relevant material.
-
Task Synergy 12: Question Answering for Open Developing Issues Introduced three years ago, Synergy envisions a continuous dialogue between biomedical experts and automated QA systems. Systems provide relevant material and answers to experts, who then assess the responses and provide feedback, including whether the material is “answer ready.” This iterative process, organized in rounds, allows systems to refine their responses using new feedback and newly available information. This year, Synergy 12 continued with four bi-weekly rounds focusing on any developing biomedical problem of interest to participating experts.
-
MultiCardioNER: Multilingual Clinical Entity Detection in Cardiology A new addition, MultiCardioNER aimed at adapting clinical entity detection to the cardiology domain in a multilingual setting, specifically Spanish, English, and Italian. It comprised two subtracks: CardioDis, focusing on disease recognition in cardiology-specific Spanish clinical case reports, and MultiDrug, which tackled multilingual medication recognition in cardiology. This task highlighted the importance of domain-specific data for improving performance in specialized medical fields.
-
BIONNE: Biomedical Nested Named Entity Recognition Also new this year, BIONNE addressed the challenge of nested named entity recognition (NER) in Russian and English PubMed abstracts. Unlike traditional NER, which identifies flat mention structures, nested NER can recognize entities within other entities, such as “[[[eye] movement] disorders].” The task included bilingual, English-oriented, and Russian-oriented tracks, pushing the state-of-the-art in complex entity extraction.
Key Trends and Achievements
The 2024 edition of BioASQ showcased several significant trends and achievements. A predominant observation was the continued and increasing reliance on deep neural approaches and Large Language Models (LLMs). Many participating systems leveraged state-of-the-art neural architectures like BERT, PubMedBERT, and BioBERT, often adapted to the specific biomedical tasks. Generative Pre-trained Transformer (GPT) models and Retrieval Augmented Generation (RAG) techniques were particularly popular, demonstrating their effectiveness in generating high-quality answers and retrieving relevant information.
Preliminary results for Task 12b indicated strong performance, especially in generating yes/no answers, with several systems achieving near-perfect scores in some batches. While there’s still room for improvement in factoid and list questions, the overall advancement is clear. The new Phase A+ also demonstrated that advanced QA approaches can perform well even without pre-selected relevant material, though access to such material still leads to improved answer quality.
In MultiCardioNER, the results underscored the critical importance of using data specific to the clinical specialty and language. Top-performing systems consistently incorporated cardiology-specific datasets, achieving higher F1-scores compared to those relying solely on general clinical texts. This suggests that even within already specialized domains, further domain adaptation is crucial for optimal performance. The multilingual aspect also revealed the need for more Italian-specific clinical models, as Spanish and English models generally performed better due to greater availability of pre-trained resources.
For BioNNE, the top-performing approaches utilized sophisticated bi-encoder frameworks with contrastive learning. The task also highlighted the limitations of pre-trained LLMs without fine-tuning, emphasizing the necessity of specialized training data for biomedical nested NER.
Also Read:
- BioASQ 2025 Unveils New Tasks in Biomedical Semantic Indexing and Question Answering
- Advancing Medical AI: A Survey of Reasoning Capabilities in Large Language Models
Looking Ahead
BioASQ 2024 reaffirmed its role as a vital platform for advancing biomedical AI. The challenge continues to expand its scope beyond English and traditional biomedical literature, incorporating new languages and more focused sub-domains. Future plans include further extending benchmark data for question answering through community-driven processes, expanding the community of biomedical experts involved in the Synergy task, and broadening the types of documents and resources considered in the challenges. For more detailed information, you can refer to the Overview of BioASQ 2024 research paper.
