TLDR: A new study evaluates the causal reasoning abilities of Large Language Models (LLMs) in interpreting laboratory tests, using Pearl’s Ladder of Causation (association, intervention, counterfactual). Testing GPT-o1 and Llama-3.2-8b-instruct on 99 clinical scenarios, the research found GPT-o1 consistently outperformed Llama-3.2, especially in complex tasks. Both models performed best on intervention questions and struggled with counterfactuals. Medically trained experts rated responses, highlighting the importance of clear, medically grounded reasoning. The study suggests LLMs show promise but require further refinement and guardrails for reliable clinical deployment.
Large Language Models (LLMs) are rapidly transforming various professional fields, including medicine, where they are increasingly used for tasks like differential diagnosis, discharge summary generation, and lab test interpretation. While these models often achieve high accuracy on standardized benchmarks, a critical area of ongoing research is their ability to perform causal reasoning – understanding and inferring cause-and-effect relationships, which is fundamental to clinical decision-making.
A recent study titled Evaluation of Causal Reasoning for Large Language Models in Contextualized Clinical Scenarios of Laboratory Test Interpretation, authored by Balu Bhasuran, Mattia Prosperi, Karim Hanna, John Petrilli, Caretia JeLayne Washington, and Zhe He, delves into this crucial aspect. The research systematically evaluates how well LLMs can reason causally in the context of interpreting common laboratory tests.
Understanding Causal Reasoning in LLMs
The study adopted Judea Pearl’s Ladder of Causation, a widely recognized framework that outlines three levels of causal reasoning:
- Association (Rung 1): Identifying statistical correlations between variables without necessarily implying cause. For example, observing that older, obese smokers tend to have higher HbA1c levels.
- Intervention (Rung 2): Deliberately changing one variable to observe its impact on another, allowing for a more direct causal interpretation. An example would be evaluating how quitting smoking affects HbA1c levels.
- Counterfactual (Rung 3): Imagining what would have happened under different conditions, such as asking if an HbA1c level would have been lower if a patient had never smoked. This level requires considering hypothetical scenarios and unmeasured factors.
Study Design and Models Evaluated
The researchers developed 99 clinically grounded laboratory test scenarios, mapping them to these three rungs of causation. The scenarios focused on eight common blood tests: Hemoglobin A1c (HbA1c), Creatinine, Vitamin D, C-reactive protein (CRP), Cortisol, Low-Density Lipoprotein (LDL), High-Density Lipoprotein (HDL), and Albumin. These were paired with clinically relevant causal factors like age, gender, obesity, smoking, physical activity, medication use, and inflammation.
Two prominent LLMs were tested: GPT-o1 (a high-performing proprietary model) and Llama-3.2-8b-instruct (a leading small-scale open-source alternative). Their responses were then independently rated by four medically trained human experts for both the correctness of the answer and the clinical validity and reliability of the reasoning provided.
Key Findings and Performance Differences
The study revealed that GPT-o1 consistently demonstrated superior discriminative performance and alignment with expert consensus compared to Llama-3.2-8b-instruct. GPT-o1 achieved an overall AUROC (Area Under the Receiver Operating Characteristic) of 0.80, compared to Llama-3.2’s 0.73. It also showed higher precision, sensitivity, and specificity across the board, particularly excelling in the more complex interventional and counterfactual reasoning scenarios.
Both models performed best on intervention questions, which often involve widely known clinical relationships (e.g., the effect of quitting smoking on HbA1c). However, both struggled most with counterfactual questions, especially those involving “altered outcome” scenarios, where agreement among human raters was also lowest.
Expert raters emphasized that high-quality responses not only provided the correct answer but also offered well-reasoned explanations grounded in medical knowledge, clinical mechanisms, and consideration of confounders like age, gender, or comorbidities. Vague or inconsistent reasoning, even with a correct final answer, led to lower scores.
For instance, in questions about creatinine levels and kidney function, GPT-o1 consistently provided correct answers and clear reasoning across all causal levels. In contrast, Llama-3.2 showed more variable performance, sometimes relying too heavily on external data without strong patient-specific connections, or offering vague explanations for association and counterfactual questions.
Also Read:
- Beyond Correlation: How AI Training Methods Shape Causal Reasoning in Language Models
- EHR-MCP: Bridging Large Language Models with Electronic Health Records
Implications and Future Directions
The findings suggest that while LLMs hold significant potential for causal reasoning in medicine, their performance is uneven and highly sensitive to how questions are framed and the clinical nuances involved. They appear more reliable for well-established interventional knowledge but less stable for complex counterfactual reasoning that requires disentangling multiple interacting causes.
The researchers recommend that future work should expand beyond just lab tests to include other clinical data like physical exam findings, imaging, and genomics. Crucially, integrating “guardrails” such as retrieval-augmented generation (RAG) with clinical guidelines, explicit disclosure of assumptions, and automatic uncertainty tagging could significantly improve model reliability and trustworthiness in real-world clinical settings. Evaluations should also move beyond simple correctness to assess whether models can identify appropriate confounders, temporal ordering, and effect modifiers in their reasoning.
This study underscores the ongoing need for refinement and careful evaluation of LLMs before their widespread deployment in high-stakes healthcare environments, particularly for tasks requiring sophisticated causal inference.
