TLDR: A study evaluated GPT-4o’s capabilities for rare disease Named Entity Recognition (NER) in low-resource settings using various prompt-based strategies. It found that structured prompting and semantically guided few-shot learning significantly improved performance, often surpassing traditional supervised models like BioClinicalBERT for rare disease and general disease entities. Task-level fine-tuning achieved new state-of-the-art results. Retrieval-augmented generation (RAG) offered limited additional benefit. The research highlights the cost-efficiency of prompt-optimized LLMs as scalable alternatives for biomedical NER, especially where annotated data is scarce, while also identifying common error types for future refinement.
Rare diseases, though individually uncommon, collectively affect hundreds of millions of people worldwide. Diagnosing and managing these conditions is challenging due to their low prevalence, varied clinical presentations, and a scarcity of readily available, structured information. This often leads to prolonged diagnostic journeys for patients. A crucial tool in addressing this gap is Named Entity Recognition (NER), an automated information extraction technique that can identify and categorize specific entities like diseases, signs, and symptoms from unstructured medical texts. However, traditional NER systems for rare diseases face significant hurdles, primarily the limited availability of high-quality, expert-annotated data, the semantic ambiguity between different entity types, and the ‘long-tail’ distribution where most rare diseases appear infrequently in data.
A recent study explores the potential of Large Language Models (LLMs), specifically GPT-4o, to overcome these challenges in rare disease NER, especially in low-resource settings. The research evaluates various prompt-based strategies, including zero-shot prompting, few-shot in-context learning, retrieval-augmented generation (RAG), and task-level fine-tuning. The goal was to assess if these advanced LLMs could offer effective and scalable alternatives to conventional supervised models.
Structured Prompting and In-Context Learning
The study designed a structured prompting framework that incorporates domain-specific knowledge and rules to help GPT-4o distinguish between four key entity types: rare disease, disease, sign, and symptom. This framework includes a clear task description, a standardized output format, detailed task guidance (definitions for each entity), and disambiguation rules to prevent common errors. Even without any labeled examples (zero-shot learning), this structured prompting significantly improved the model’s ability to recognize entities, particularly enhancing recall for rare diseases.
Further improvements were observed with ‘few-shot in-context learning,’ where a small number of labeled examples are provided directly within the prompt. The researchers found that the selection of these examples was critical. Strategies that picked examples based on semantic similarity to the inquiry text (like Inquiry-KNN and Cluster-KNN) consistently outperformed random selection. For rare disease and general disease entities, few-shot learning with semantically aligned examples even surpassed the performance of BioClinicalBERT, a specialized biomedical model that requires extensive training data. This highlights the efficiency of LLMs in learning from just a handful of relevant examples, significantly reducing the need for costly manual annotation.
Fine-Tuning for State-of-the-Art Performance
While prompt-based methods showed strong results, the study also investigated ‘task-level fine-tuning,’ where the LLM’s parameters are updated using the RareDis Corpus training set. This approach allowed GPT-4o-mini to internalize task-specific patterns. The fine-tuned GPT-4o-mini achieved new state-of-the-art (SOTA) results for rare disease, disease, and sign entities, outperforming both the prompt-based methods and BioClinicalBERT. This demonstrates that when task-specific labeled data is available, full model fine-tuning can yield superior performance, especially in terms of balanced precision and recall.
The Limited Role of Retrieval-Augmented Generation (RAG)
Interestingly, the study found that Retrieval-Augmented Generation (RAG), which allows the LLM to access external knowledge snippets at inference time, provided only marginal benefits. This suggests that GPT-4o already possesses a substantial amount of biomedical knowledge from its pre-training, and short RAG snippets often added little new information. This finding contrasts with RAG’s significant impact in other knowledge-intensive tasks, indicating that for rare disease NER, the challenge is more about precise span recognition within context rather than a lack of general world knowledge.
Also Read:
- Adaptive Prompting Strategies Boost Biomedical Named Entity Recognition
- GPT-5 Achieves Breakthrough in Multimodal Medical Reasoning, Outperforming Human Experts
Cost-Efficiency and Future Outlook
A key takeaway from the research is the impressive cost-efficiency of prompt-based LLMs. Few-shot prompting, in particular, delivers high returns at a low token budget, making it an attractive solution for low-resource NER scenarios where extensive data annotation is not feasible. For rare disease, disease, and sign entities, a small investment in prompt tokens (equivalent to up to 8-shot prompting) was sufficient to capture most of the attainable performance gains. The study also provided an error analysis, identifying common failure modes such as boundary mismatches and spurious (false positive) recognitions, especially for signs and symptoms. These insights suggest opportunities for post-processing and hybrid systems to further refine LLM outputs.
This research demonstrates that thoughtfully engineered prompts and semantically selected learning examples can make LLMs powerful tools for rare disease NER. The ability to achieve near-SOTA results with minimal labeled data offers a compelling solution for institutions lacking extensive machine learning infrastructure. Such LLM-powered NER systems could be integrated into electronic health records, assist in biomedical knowledge curation, and be easily adapted for new subtasks by simply modifying prompts. For more technical details, you can refer to the full research paper: Leveraging Large Language Models for Rare Disease Named Entity Recognition.
