TLDR: The MasonNLP team at MEDIQA-OE 2025 demonstrated that large, general-purpose language models (LLaMA-4 17B), without domain-specific fine-tuning, can effectively extract structured medical orders from complex doctor-patient conversations. By using few-shot prompt engineering, their system achieved competitive results, particularly in identifying order types, descriptions, reasons, and provenance. The study highlights the potential of instruction-tuned LLMs as scalable baselines for clinical NLP tasks, while also pointing out challenges in extracting implicit reasons and precise provenance.
Medical order extraction is a crucial process in healthcare, transforming unstructured clinical information into actionable data. This structured data supports vital functions like decision-making, documentation, and automating workflows within healthcare systems. Medical orders can originate from various sources, including electronic health records, discharge summaries, and even doctor-patient conversations, covering categories such as medications, laboratory tests, imaging studies, and follow-up actions.
The MEDIQA-OE 2025 shared task specifically challenged participants to extract structured medical orders from extended conversational transcripts. This task required identifying the order type, a detailed description, the reason or justification for the order, and its provenance (where in the conversation the order was mentioned).
A team from George Mason University, MasonNLP, participated in this challenge, presenting an innovative approach that secured 5th place among 17 participating teams. Their method stood out because it utilized a general-purpose, instruction-tuned LLaMA-4 17B model without any specialized fine-tuning for the medical domain. Instead, their strategy relied on effective prompt engineering, guided by a single in-context example, a technique known as few-shot learning.
This few-shot configuration achieved an average F1 score of 37.76, demonstrating significant improvements in the accuracy of extracting the reason and provenance of medical orders. These results highlight a key finding: large language models, even those not specifically trained on medical data, can serve as strong and scalable foundational tools for specialized clinical natural language processing tasks when paired with well-designed prompts.
The MasonNLP Approach
The MasonNLP team’s methodology involved converting input transcripts from a JSON format into plain text, preserving turn order and speaker roles. They evaluated three different configurations of Meta-Llama models:
- LLaMA-3 8B in a zero-shot setting (no in-context examples).
- LLaMA-4 17B in a zero-shot setting.
- LLaMA-4 17B in a few-shot setting, which included a single example from the training set to guide the model.
The prompt design was critical. It defined the model’s role as a clinical assistant and explicitly outlined the required output structure, including allowed values for order types (medication, lab, imaging, followup). The team iteratively refined their prompts, moving from free-form text to a more structured, comma-separated line format that the model followed more consistently, ensuring outputs were grounded and easy to parse.
Performance and Insights
The ablation study conducted by MasonNLP clearly showed a performance increase with larger models and the introduction of few-shot prompting. The LLaMA-4 17B few-shot configuration consistently outperformed the zero-shot versions across all subtasks, with the most notable gains in provenance extraction. This suggests that even minimal task-specific guidance significantly helps the model in grounding predictions and adhering to the required structured format.
Despite the strong performance, the research identified areas for improvement. Extracting the ‘reason’ for an order remained the most challenging subtask, often due to the implicit nature of clinical justifications in conversations. Provenance accuracy also lagged, indicating difficulty in linking orders to scattered or indirect evidence within the dialogue. The model sometimes omitted finer details like exact timing or numeric values and occasionally produced ‘hallucinations’ (text not present in the transcript).
Also Read:
- Advancing Biomedical Entity Recognition with a Unified LLM Framework
- Unlocking Complex Skills: How AI Bridges the Granularity Gap in Competency Modeling
Future Directions
The study concludes that general-domain instruction-tuned LLMs, when combined with carefully designed prompts and minimal in-context examples, can achieve competitive results in structured clinical information extraction. Future enhancements could involve integrating retrieval-based grounding (RAG) to reduce hallucinations and improve evidence alignment, or further domain adaptation to better handle implicit reasoning and balance performance across different order types.
For more in-depth information, you can read the full research paper here.
