TLDR: A study on LLMs’ causal understanding reveals that exposure to training data doesn’t improve their ability to identify causal relationships, nor do they show memorization bias. Base models are appropriately uncertain, but instruction-tuned models exhibit severe overconfidence despite low accuracy. Failures in causal understanding are attributed to a lack of structured causal representation, especially with conditional relationships, rather than insufficient data exposure.
Large Language Models (LLMs) have shown remarkable capabilities in various natural language tasks, but their ability to truly grasp causal relationships from text has been a persistent challenge. A recent research paper, “Causal Understanding by LLMs: The Role of Uncertainty,” delves into why these powerful models struggle with causality, investigating whether the issue lies in insufficient training exposure or more fundamental representational gaps.
Authored by Oscar Lithgow-Serrano, Vani Kanjirangat, and Alessandro Antonucci from SUPSI, IDSIA, Switzerland, the study explores LLM behavior through the lens of uncertainty. Previous work indicated that LLMs perform almost randomly in classifying causal relations. This new research aims to understand if exposure to causal examples during pretraining improves this understanding, particularly by analyzing how models express their uncertainty.
Investigating Causal Understanding and Memorization
The researchers designed controlled experiments using over 18,000 PubMed sentences. Half of these sentences were from The Pile corpus, which many LLMs were trained on, making them “seen” data. The other half were post-2024 sentences, considered “unseen” by the models. Seven different LLMs were tested, including Pythia variants (1.4B, 7B, 12B), GPT-J-6B, Dolly-7B/12B, and Qwen-7B.
Two primary tasks were used:
Causal Classification: Models had to identify causal relationships in text, classifying them into four categories: direct causal, conditional causal, correlational, or no relationship. This directly tested their understanding.
Verbatim Memorization Probing: This task assessed whether models preferred previously seen causal statements over their paraphrases. The idea was to see if models simply memorized surface forms rather than truly understanding the underlying causal patterns.
Key Findings: Exposure Isn’t Enough, Overconfidence is a Concern
The study yielded several crucial insights into LLMs’ causal understanding:
No Memorization Advantage: Models showed almost identical accuracy on both seen and unseen sentences. This suggests that simply exposing LLMs to causal examples during pretraining does not reliably lead to better performance or verbatim recall. The models did not leverage previously observed patterns for causal tasks.
Random Guessing with Appropriate Uncertainty (for some): Many models, particularly the base pretrained ones like Pythia and GPT-J, performed near random chance (25% accuracy for a four-way classification) but exhibited high entropy, meaning they were appropriately uncertain about their predictions. Their output distributions were almost flat, confirming random guessing.
Instruction Tuning Breeds Overconfidence: A significant finding was the behavior of instruction-tuned models like Qwen and Dolly. While Qwen-7B-base achieved slightly better accuracy (32.8%), it did so with severe miscalibration, assigning over 95% confidence to its predictions. This confidence-accuracy gap resulted in a much higher Expected Calibration Error (ECE) compared to base models. Dolly models also showed moderate confidence without corresponding accuracy improvements. This suggests that instruction tuning teaches models to be confident, even when that confidence is unjustified, which can be problematic in critical applications.
Causal Complexity Increases Uncertainty: The models struggled most with “conditional causal” relationships, which involve nuances like conditions or moderating factors. These statements induced the highest entropy (uncertainty) across all models, indicating a lack of nuanced causal representations rather than just insufficient data exposure.
Ambiguity in Questions: The analysis also revealed that a significant portion of questions led to consistent incorrect answers across paraphrases, suggesting inherent ambiguity in the task itself, contributing to aleatoric uncertainty.
Also Read:
- The Inherent Design Flaw: How Transformers Generate Coherence Without Truth
- Understanding and Addressing Hallucinations in AI Agents
Implications for Future LLM Development
The authors conclude that failures in causal understanding stem from a lack of structured causal representation within LLMs, rather than merely insufficient exposure to causal examples during pretraining. Simply scaling up datasets is unlikely to solve these deficits. Instead, architectural innovations or training objectives that explicitly target causal inference are needed.
The paper also highlights the risks associated with overconfidence in instruction-tuned models, especially in domains requiring reliable causal inference. This shift from appropriately calibrated uncertainty in base models to overconfidence in fine-tuned ones is a critical area for future research and development. For more details, you can read the full research paper available at arXiv.
