TLDR: The EMRC framework enhances medical decision-making by dynamically selecting specialized Large Language Models (LLMs) based on their expertise for specific medical queries and then integrating their responses through a confidence- and adversarial-driven collaboration process. This two-stage approach significantly improves diagnostic accuracy and reliability compared to single LLMs and other multi-agent methods.
Medical decision-making (MDM) is a highly intricate process that demands deep specialized knowledge to accurately interpret complex clinical information. While Large Language Models (LLMs) have shown great potential in supporting MDM, individual LLMs often struggle due to their limited knowledge and static training data, making it difficult for them to fully integrate diverse clinical information.
To overcome these challenges, researchers have introduced a novel framework called Expertise-aware Multi-LLM Recruitment and Collaboration (EMRC). This innovative system aims to significantly improve the accuracy and reliability of AI-assisted medical diagnoses. The EMRC framework operates in two main stages, designed to leverage the unique strengths of multiple LLMs.
Expertise-aware Agent Recruitment
The first stage focuses on intelligently selecting the most suitable LLMs for a given medical query. Unlike previous approaches that might treat all LLMs as equally capable or assign generic roles, EMRC recognizes that different LLMs possess specialized strengths across various medical domains and query difficulty levels. To achieve this, the framework constructs an “LLM expertise table.” This table systematically quantifies how well each LLM performs in different medical departments (like Internal Medicine, Surgery, Pediatrics, etc.) and at various difficulty levels (low, medium, high).
During the inference phase, when a new medical query comes in, a dedicated query classifier (itself an LLM chosen for its classification proficiency) first determines the query’s medical department and difficulty. Based on this classification, the EMRC framework then dynamically selects a subset of LLMs from its pool that have demonstrated the highest expertise in that specific area and difficulty level. This ensures that the most knowledgeable “medical expert agents” are recruited for each unique case.
Confidence- and Adversarial-Driven Multi-agent Collaboration
The second stage addresses the critical issue of ensuring consistency and accuracy when integrating information from multiple LLMs, especially given the risk of “hallucinations” (where LLMs generate plausible but incorrect information). EMRC employs a sophisticated multi-layer collaboration architecture to refine responses iteratively.
Initially, each recruited LLM generates its own response to the medical query, along with a self-assessed confidence score. This self-assessed score is then combined with the LLM’s historical expertise score (from the expertise table) to create an overall confidence score for its answer. Simultaneously, the LLM with the highest expertise in the relevant category is designated as the “Judge.” This Judge performs an adversarial verification, cross-checking all candidate responses for factual inconsistencies, errors, or contradictions. This process generates valuable feedback to guide refinement.
The outputs from this first round—including confidence scores, identified errors, and initial responses—are then passed to subsequent collaboration layers. In these layers, agents refine their outputs based on the integrated feedback. Finally, a designated “Aggregator” LLM synthesizes all the refined information to produce a single, high-quality, and reliable final diagnosis. This iterative refinement process significantly enhances the accuracy and trustworthiness of the medical decision-making output.
Also Read:
- Advancing Language Model Reasoning Through Collaborative Learning
- Building a Dynamic Medical Knowledge Graph with AI Agents
Demonstrated Superior Performance
The effectiveness of the EMRC framework has been rigorously tested on three public medical decision-making datasets: MedQA, NEJMQA, and MMLU-Pro-Health. The results consistently show that EMRC outperforms both state-of-the-art single LLMs and other multi-agent collaboration methods. For instance, on the MMLU-Pro-Health dataset, EMRC achieved an impressive 74.45% accuracy, surpassing the best-performing closed-source model, GPT-4-0613, by 2.69%.
Ablation studies further confirmed the importance of EMRC’s core components. The expertise-aware agent recruitment strategy proved superior to random or task-level top-N selections, highlighting the value of domain-specific LLM selection. Similarly, the confidence- and adversarial-driven collaboration mechanism significantly improved performance, demonstrating the necessity of robust information integration and error correction. The research also explored the optimal number of recruited agents and collaboration layers, finding that four agents and two layers yielded the best results, balancing diversity with quality and avoiding diminishing returns.
In conclusion, the EMRC framework represents a significant advancement in AI-assisted medical decision-making. By intelligently recruiting LLMs based on their specific expertise and fostering a robust, confidence- and adversarial-driven collaboration, it sets a new standard for diagnostic accuracy and reliability in healthcare. You can find more details about this research in the full paper available here.
