TLDR: Skin-SOAP is a new weakly supervised multimodal framework that generates structured SOAP (Subjective, Objective, Assessment, Plan) notes from limited inputs like lesion images and sparse clinical text. It aims to reduce the labor-intensive process of manual note-taking, which contributes to clinician burnout. The framework uses a novel approach involving generative captioning, retrieval-augmented knowledge integration, and fine-tuning a Vision-LLaMA model. Evaluations show Skin-SOAP performs comparably to or better than state-of-the-art LLMs like GPT-4o in clinical relevance and coherence, offering a scalable solution for dermatology documentation.
Skin cancer is a global health concern, being the most common form of cancer and incurring significant healthcare costs. Early and accurate diagnosis, along with timely treatment, are crucial for improving patient outcomes. In clinical practice, doctors meticulously document patient visits using detailed SOAP (Subjective, Objective, Assessment, and Plan) notes. However, the manual creation of these notes is a labor-intensive process that often contributes to clinician burnout.
Addressing this challenge, researchers have introduced Skin-SOAP, a novel framework designed to automate the generation of structured SOAP notes. This innovative system is particularly noteworthy because it operates effectively with limited inputs, specifically lesion images and sparse clinical text. Unlike many existing methods that demand extensive manual annotations or large, pre-existing datasets, Skin-SOAP significantly reduces this reliance, making it a scalable solution for clinical documentation.
Previous attempts at automating SOAP note generation, such as K-SOAP, often depend heavily on large volumes of doctor-patient dialogues and annotated datasets. These resources are particularly scarce in specialized fields like dermatology. Furthermore, integrating both the visual characteristics of skin conditions and the underlying clinical reasoning into a structured format has been a major hurdle. General-purpose large language models (LLMs), while powerful, frequently lack the specific medical reasoning required for clinical settings and are typically limited to text-based inputs, struggling with structured note generation in domains like dermatology.
Skin-SOAP overcomes these limitations by employing a weakly supervised multimodal approach. It uniquely integrates retrieval-augmented clinical knowledge, weak supervision, and multimodal synthesis. This allows it to generate domain-aligned documentation without the need for extensive, large-scale annotations. The framework operates in three main phases: data generation, fine-tuning, and inference.
In the data generation phase, Skin-SOAP leverages the PAD-UFES-20 dataset, which includes dermoscopic images and structured metadata for various skin lesions. To compensate for the lack of large annotated SOAP note datasets, the system uses GPT-3.5 to create clinical captions from structured patient attributes. These captions are then used as queries to retrieve relevant medical information from a curated database, built from authoritative sources like the National Cancer Institute and the American Cancer Society. This retrieval-augmented generation (RAG) approach helps ensure the clinical relevance and factual reliability of the generated notes, guiding a pre-trained Vision-LLaMA 3.2 model to produce weakly supervised SOAP notes in the correct format.
The fine-tuning phase adapts the Vision-LLaMA 3.2 model using these synthesized notes. The model learns to map multimodal inputs (lesion images and generated captions) to structured SOAP notes. To optimize computational costs, the researchers employed Parameter-Efficient Fine-Tuning (PEFT) strategies, specifically Quantized Low-Rank Adaptation (QLoRA), which efficiently updates the model without requiring full parameter changes.
During inference, the fine-tuned Vision-LLaMA model takes a lesion image and its clinical features (converted into a caption) and generates a structured SOAP note. Because the model is trained on clinically reliable, albeit weakly supervised, data, it can effectively generalize to new cases, enabling scalable and structured documentation even when expert annotations are scarce.
The effectiveness of Skin-SOAP was rigorously evaluated using both quantitative and qualitative methods. The researchers introduced two novel clinical relevance metrics: MedConceptEval and Clinical Coherence Score (CCS). MedConceptEval assesses the semantic alignment of each SOAP note section with clinically validated concept sets, ensuring the notes align with disease-specific medical terminology. The Assessment and Plan sections consistently showed higher alignment, particularly for conditions like Melanoma and Nevus.
The Clinical Coherence Score (CCS) evaluates the semantic alignment between the initial clinical caption and the generated SOAP note sections. Interestingly, the LLM-generated SOAP notes exhibited consistently higher semantic alignment with the captions compared to dermatologist-written notes. While this suggests the model is adept at capturing terminology from the input, it also highlights an area for future research: bridging the gap between the model’s output and the depth of clinical reasoning often found in human-written notes.
In comparative evaluations against leading models like GPT-4o, Claude, and DeepSeek Janus Pro, Skin-SOAP demonstrated strong performance. It particularly excelled in metrics like METEOR and CHRF++, indicating high fluency and surface-level coherence. Crucially, its ClinicalBERT F1 scores were either the highest or on par with the best-performing models, underscoring its superior alignment with clinical concepts. A qualitative evaluation using an LLM-as-a-Judge framework (Flow-Judge-v0.1) further confirmed Skin-SOAP’s capabilities, with the framework achieving a perfect score for structure, readability, completeness, and medical relevance, outperforming other state-of-the-art models.
While Skin-SOAP shows immense promise for generating structured SOAP notes, the researchers acknowledge certain limitations. The quality of the generated notes is dependent on the accuracy of the retrieved domain-specific knowledge, and the evaluation was constrained by a small set of expert-annotated samples. Like many generative models, there’s a risk of hallucination with ambiguous inputs, though retrieval-augmented generation helps mitigate this. Future work will focus on expanding to more diverse datasets and incorporating human-in-the-loop refinement strategies to further enhance the system’s utility in real-world healthcare settings.
Also Read:
- Advancing Radiology Report Generation with a New Multi-modal Knowledge Graph
- RegionMed-CLIP: A Smarter Approach to Medical Image Understanding
This framework represents a significant step forward in automating clinical documentation, with the potential to streamline dermatology workflows, reduce time-to-treatment, and ultimately enhance patient care. For more details, you can refer to the full research paper: Skin-SOAP: A Weakly Supervised Framework for Generating Structured SOAP Notes.
