TLDR: This paper investigates how well Large Language Model (LLM) uncertainty measures align with human uncertainty and predict model correctness. It finds that many entropy-based inference-time uncertainty measures, such as choice entropy and top-k entropies, strongly correlate with human group uncertainty and are also well-calibrated, suggesting a path towards more intuitive and trustworthy LLM-human interactions.
Large Language Models (LLMs) are becoming increasingly integrated into our daily lives, from answering questions to generating creative content. However, a critical challenge remains: how well do these models understand and communicate their own uncertainties? This is crucial for building user trust and preventing the spread of misinformation, especially when LLMs might “hallucinate” or provide incorrect information. A recent research paper, “Human-Alignment and Calibration of Inference-Time Uncertainty in Large Language Models”, by Kyle Moore, Jesse Roberts, and Daryl Watson, delves into this very topic, exploring how LLM uncertainty measures align with human uncertainty and how well they predict model correctness.
Understanding LLM Uncertainty
Uncertainty quantification (UQ) in LLMs is about knowing when a model is likely to be right or wrong. While much research has focused on making these uncertainty measures “calibrated” – meaning they accurately reflect the model’s likelihood of generating a valid answer – less attention has been paid to whether these measures make sense to humans. In other words, does a high uncertainty score from an LLM feel like high uncertainty to a person?
This paper specifically focuses on “inference-time uncertainty,” which means the uncertainty can be calculated in real-time as the model generates its output. This is important because it can provide an immediate signal to users or other systems, allowing for better control and interaction without significant delays.
Bridging the Gap: Human Alignment
The core innovation of this research is its focus on “human alignment.” This refers to how closely an AI system’s behavior matches human behavior. For uncertainty, it means evaluating if an LLM’s uncertainty measure correlates well with how uncertain a group of humans would be about the same question. Since measuring individual human uncertainty is difficult, the researchers approximated this by looking at the level of disagreement among human survey respondents on multiple-choice questions.
The study used two datasets: a smaller one from Pew Research surveys and a much larger one with nearly 3000 questions from the Roper Center for Public Opinion Research. These datasets provided a rich source of human responses to compare against LLM uncertainty.
Measures of Uncertainty Explored
The researchers investigated several ways to measure LLM uncertainty at inference time. One simple approach is the Top-1 Probability, which is the probability assigned to the single most likely output token. The majority of the candidate measures, however, were based on Entropy. Entropy quantifies the unpredictability or “disorder” in the model’s probability distribution over possible output tokens; higher entropy generally indicates higher uncertainty.
Variations of entropy-based measures included Total Entropy, calculated across the entire vocabulary of possible tokens. They also explored Top-k Entropy, where entropy is calculated only on the top ‘k’ most probable tokens (e.g., top 5, 10, 25, 50, or 100 tokens). For multiple-choice questions, they used Choice Entropy, which specifically calculates entropy over the normalized probabilities of the provided answer choices.
Another set of measures involved Top-p Sampling (also known as nucleus sampling), where tokens are extracted until their cumulative probability reaches a certain threshold ‘p’. The researchers investigated both the Top-p Size (the number of tokens in this set) and Top-p Entropy (the entropy over the normalized probabilities of these top-p tokens) as indicators of uncertainty.
Key Findings: Alignment and Calibration
The results were quite insightful. While LLMs showed only mild agreement with humans on the single best answer and very little alignment in their overall preference ordering of answers, the uncertainty measures told a different story.
Many entropy-based measures, including total entropy, all top-k entropies, and top-p entropy (for higher ‘p’ values), showed strong correlation with human uncertainty. This means that when humans were more uncertain about a question, these LLM measures also indicated higher uncertainty. This is a significant step towards making LLM uncertainty more intuitively understandable for users.
Furthermore, the study also evaluated the “calibration” of these measures – how well they predict the model’s correctness. Using a benchmark called MMLU (Massive Multitask Language Understanding), the researchers found that the human-aligned measures also showed evidence of calibration. Notably, “choice entropy” emerged as a strong performer, consistently showing a good correlation with the model’s correctness across various subjects.
A novel calibration metric, the Jensen-Shannon Distance Shift, was also introduced, which directly evaluates how the model’s answer distribution changes from correct to uncertain states. This analysis further supported the finding that these aligned measures are indeed well-calibrated.
Also Read:
- Addressing Overconfidence in AI Judges: New Metrics and Ensemble Approaches
- Building Safer AGI: A New Framework Merges Active Inference with Language Models
Implications and Future Directions
This research provides strong initial evidence that certain entropy-based uncertainty measures in LLMs can align well with human uncertainty, even if the models don’t always agree with human answer preferences. This has profound implications for how we interact with LLMs. If models can communicate their confidence in a way that humans intuitively understand, it could significantly increase user trust and improve the overall human-LLM experience.
Future work will explore extending these findings to more open-ended questions, investigating uncertainty alignment at the individual human level, and directly studying how human-aligned uncertainty measures benefit user experience in real-world applications. The researchers hope that integrating these intuitive confidence signals into AI agents will lead to more trustworthy and effective human-LLM collaboration.
