TLDR: A new framework called Semantic Divergence Metrics (SDM) helps detect AI hallucinations by measuring how much an LLM’s response deviates from the original prompt. Unlike older methods, SDM is “prompt-aware,” using paraphrased prompts and joint topic clustering to assess semantic instability (SH score) and semantic exploration (KL Divergence). It identifies “confident hallucinations” where models give stable but incorrect answers to nonsensical questions, and uses a “Semantic Box” to classify different LLM behaviors from factual recall to creative generation.
Large Language Models, or LLMs, have become incredibly powerful tools, capable of generating text that often sounds remarkably human. However, a significant challenge remains: hallucinations. These are instances where LLMs produce information that is non-factual, nonsensical, or unfaithful to the input they were given. This problem is critical because it undermines the reliability of these advanced AI systems.
A new research paper introduces a novel approach to tackle this issue, focusing specifically on “faithfulness hallucinations.” These occur when an LLM’s response severely deviates from the original input context. The paper, titled “Prompt-Response Semantic Divergence Metrics for Faithfulness Hallucination and Misalignment Detection in Large Language Models,” proposes a lightweight framework called Semantic Divergence Metrics (SDM).
Understanding the Problem: Hallucinations and Confabulations
Hallucinations in LLMs can be categorized in various ways. Some contradict the provided input (intrinsic), while others cannot be verified against external knowledge (extrinsic). This paper primarily focuses on intrinsic faithfulness hallucinations, which they often refer to as “confabulations.” In simple terms, a confabulation is an LLM output that is both incorrect and arbitrary, semantically misaligned with the user’s query. Think of it as the model filling in gaps with fabricated details, much like how a person might confabulate false memories without intending to deceive.
Existing methods, like Semantic Entropy, try to detect this arbitrariness by measuring how diverse an LLM’s answers are to a single, fixed prompt. If the model gives many different answers to the same question, it might be seen as unstable. However, a key limitation of these methods is that they are not “prompt-aware.” They don’t fully consider the context or complexity of the original prompt, potentially flagging a legitimately complex and varied answer as a hallucination.
Introducing Semantic Divergence Metrics (SDM)
The SDM framework improves upon this by being more sophisticated and prompt-aware. Its core idea is to measure response consistency not just across multiple answers, but also across multiple, semantically equivalent paraphrases of the original prompt. This means they test for a deeper form of arbitrariness. The framework operates in a “black-box” setting, meaning it only needs the model’s inputs and outputs.
Methodologically, SDM uses a clever technique involving “joint clustering” on sentence embeddings. Imagine taking all the sentences from both the prompts and the LLM’s answers and grouping them into shared “topic spaces.” This creates a quantified, two-dimensional visualization of the user-machine dialogue, like a heatmap showing how topics from prompts and responses co-occur.
From this shared topic space, the researchers compute a suite of metrics to measure the “semantic divergence” between prompts and responses. Two key metrics emerge:
- SH Score (Semantic Instability): This practical score combines two measures to quantify semantic drift. A high SH score indicates a higher risk of a faithfulness hallucination. It tells us how much the answers deviate in meaning from the prompts.
- KL Divergence (Semantic Exploration): This metric is a powerful indicator of “Semantic Exploration.” It helps distinguish different generative behaviors, showing how much the LLM has to “explore” or invent new semantic content beyond what was explicitly provided in the prompt.
These metrics are then combined into a diagnostic tool called the “Semantic Box,” which helps classify different types of LLM responses, including the dangerous “confident confabulation.”
Experiments and Key Findings
The researchers conducted two sets of experiments using the GPT-4o model to validate their framework. The first set tested the framework’s sensitivity across a “stability gradient,” from factual questions to highly creative tasks. The second set used diverse prompt types, including a deliberately nonsensical one to force a hallucination.
One of the most striking findings was related to the “Forced Hallucination” prompt (e.g., asking about the influence of Quantum Chromodynamics on Baroque music). Counter-intuitively, this prompt yielded the *lowest* SH score. This doesn’t mean the answer was factually correct; it means the model, when faced with a nonsensical query, produced a highly stable and consistent “evasion strategy” rather than random or unstable outputs. This “confident hallucination” is a critical failure mode where the model fabricates a stable falsehood instead of admitting uncertainty.
The experiments also highlighted the role of KL Divergence. For prompts that required the model to “invent” or “explore” new semantic content (like a creative writing task), the KL Divergence values were significantly higher. For prompts that were “conceptually contained” (like factual summaries), the KL Divergence was much lower. This shows that KL Divergence is not just a measure of instability, but a sensitive indicator of the “cognitive work” of interpretation and synthesis required by the prompt.
The study also confirmed the limitations of older, prompt-agnostic methods like Semantic Entropy, which often incorrectly flagged diverse but correct answers as problematic because they didn’t account for the prompt’s inherent complexity.
The Semantic Box: Classifying LLM Behavior
To provide a clearer interpretation of LLM responses, the Semantic Box framework categorizes behavior based on the SH score (Semantic Instability) and KL Divergence (Semantic Exploration). It divides responses into four distinct zones:
- Faithful Factual Recall (Green Box): Low exploration, some minor instability. Ideal for factual queries where the model synthesizes real-world facts.
- Faithful Interpretation (Yellow Box): High exploration, low instability. Ideal for interpretive tasks where the model consistently elaborates on defined themes.
- Creative Generation (Orange Box): High instability, high exploration. Expected for purely creative tasks where the model explores a wide and novel topic space.
- Convergent Response (Red Box): Low instability, low exploration. This is the ambiguous zone. It can be a benign “trivial echoic response” for very simple prompts, or, more dangerously, a “confident hallucination” where the model produces a stable but incorrect answer template for a difficult or nonsensical query.
This framework moves beyond a simple pass/fail score, offering a nuanced diagnostic tool for understanding LLM behavior.
Also Read:
- Rethinking How We Measure AI Hallucinations
- Evaluating LLM Alignment: A New Framework for Safety and Performance
Looking Ahead
The SDM framework represents a significant step towards building more reliable and interpretable evaluation systems for LLMs. Future work includes large-scale validation, developing a “self-calibrating” system that can adapt to different prompt types on the fly, and applying the framework to measure “semantic groundedness” in systems like Retrieval-Augmented Generation (RAG), where LLMs must stick to provided source documents.
For more technical details, you can refer to the full research paper available at arXiv:2508.10192.
