TLDR: The research introduces the Healthcare Agent Orchestrator (HAO), an AI agent system that uses Large Language Models to create accurate and comprehensive patient summaries for Molecular Tumor Boards (MTBs), addressing the current labor-intensive and subjective manual process. It also presents TBFact, a “model-as-a-judge” framework for evaluating the quality of these AI-generated summaries, ensuring completeness and succinctness. HAO, even in single-agent mode, significantly improves information quality and efficiency in MTB preparations.
Molecular Tumor Boards (MTBs) are crucial multidisciplinary meetings where cancer specialists come together to discuss complex patient cases and decide on the best treatment plans. A key part of these discussions is the patient summary, which is usually put together manually by medical professionals. This process involves sifting through a vast amount of diverse medical records to create a clear, concise narrative. However, this manual approach is often time-consuming, can be subjective, and sometimes misses critical information, with preparation times for radiologists and pathologists often exceeding an hour for each case.
To tackle these challenges, researchers have introduced the Healthcare Agent Orchestrator (HAO). HAO is an AI agent system powered by Large Language Models (LLMs) designed to automate and improve the creation of patient summaries for MTBs. Instead of relying on a single, all-encompassing AI model, HAO coordinates multiple specialized AI agents, each focusing on a specific domain like patient history, radiology, pathology, or clinical guidelines. This modular design mimics the collaborative nature of real tumor boards, allowing for precise reasoning across different data sources while maintaining transparency and explainability.
HAO is built around three core principles: precision through specialization, traceability with shared memory and inline citations, and safety-by-design through verification checkpoints. This architecture helps reduce errors and ensures that the AI system’s conclusions are grounded in evidence. The user experience for HAO is integrated directly into Microsoft Teams, allowing clinicians to interact with the orchestrator or specific agents within their existing communication channels. Outputs, such as patient timelines or draft MTB briefs, can then be easily distributed across Microsoft 365 applications like Word and PowerPoint, streamlining the workflow for preparing tumor board packets.
For example, a clinical assistant can use the Patient History agent alone to quickly generate a concise, citation-backed timeline for an MTB opening brief. For more complex cases, the orchestrator can engage multiple agents in sequence to produce a comprehensive report, providing flexibility for diverse information needs.
Evaluating the quality of these AI-generated patient summaries presents its own set of difficulties. Traditional similarity metrics often fail to capture clinically meaningful differences, as summaries might use different phrasing or ordering but convey the same facts, or conversely, appear similar while omitting crucial details. To address this, the researchers developed TBFact, an evaluation framework specifically designed to assess the comprehensiveness and succinctness of generated summaries at the level of clinical factual claims.
TBFact’s evaluation process involves four main stages: extracting clinical factual claims from both reference and candidate summaries, classifying each fact by its clinical importance (high, medium, low), determining bidirectional entailment (whether facts are fully, partially, or not entailed by the counterpart text), and attributing errors as omissions or unsupported claims. This framework allows for a detailed and clinically relevant assessment of summary quality, even providing partial credit for incremental improvements.
Using a de-identified dataset called TB-Bench, which includes longitudinal materials for 71 oncology patients, the Patient History agent was evaluated. Results showed that with specialized prompting, the agent achieved a TBFact recall of 0.84 on high-importance facts under strict entailment criteria. This means it captured 94% of high-importance information when partial entailments were considered. This performance suggests that the single Patient History agent can match the information quality of current manual MTB summaries, potentially saving significant preparation time without sacrificing completeness.
A human validation study further confirmed TBFact’s reliability. Medically trained annotators showed near-perfect agreement (0.999) on claim extraction validity, high agreement on clinical importance classification (93% allowing for one-level difference), and strong agreement on entailment judgments (88%). Critically, end-to-end TBFact F1 scores showed strong correlations with human expert assessments, validating TBFact as an effective proxy for expert evaluation of clinical factuality and completeness. This framework also allows institutions to deploy it locally for evaluation without sharing sensitive clinical data, aligning with clinical governance requirements.
Also Read:
- Advancing Medical Decision Support with Integrated Patient Data and AI
- HyFedRAG: A New Framework for Private and Diverse Data in AI
The HAO and TBFact systems offer a practical foundation for improving MTB preparation, providing reliable and scalable support. While the current findings focus on a single specialized agent, there is clear potential for future extensions to multi-agent, multi-modal workflows and further refinements of the evaluation metrics. You can read the full research paper here: Demo: Healthcare Agent Orchestrator (HAO) for Patient Summarization in Molecular Tumor Boards.
