TLDR: This research introduces Generalized Correctness Models (GCMs), a new approach to predict whether a Large Language Model’s (LLM) answer is correct. Contrary to common assumptions, the study finds that LLMs have little “self-knowledge” about their own correctness; one LLM is generally no better at predicting its own accuracy than another. GCMs address this by learning from the historical performance of *multiple* LLMs, enabling them to develop generalizable strategies for correctness prediction. These models demonstrate superior accuracy and calibration across various LLMs and datasets, even outperforming larger models’ self-reported confidences. The research also highlights the importance of answer phrasing and world knowledge in prediction and shows how GCMs can enhance downstream tasks like selective prediction, offering higher coverage with lower risk.
In the rapidly evolving landscape of Large Language Models (LLMs), ensuring their reliability and trustworthiness is paramount, especially for critical applications. A key aspect of this reliability is the ability of an LLM to accurately estimate its own confidence in the answers it provides. This concept, often referred to as ‘confidence calibration,’ ensures that when an LLM states it’s 90% confident, its answer is indeed correct 90% of the time.
Traditional approaches to LLM confidence have largely focused on eliciting a model’s ‘self-knowledge’ – essentially asking an LLM to judge the correctness of its own outputs. The implicit assumption here is that the model possesses some unique insight into its internal processes that makes it best suited to assess its own accuracy. However, recent research challenges this very assumption.
Challenging the Notion of LLM Self-Knowledge
A groundbreaking study reveals that this long-held belief about LLM self-knowledge might not hold true. Experiments conducted by researchers indicate that an LLM attempting to predict the correctness of its own answers performs no better than an entirely unrelated LLM performing the same task. This suggests that LLMs have very little inherent ‘self-knowledge’ when it comes to predicting their own correctness. In essence, using a stronger, more knowledgeable LLM to predict correctness is more beneficial than relying on the generating model itself.
Introducing Generalized Correctness Models (GCMs)
This pivotal finding opens the door to a new paradigm: instead of focusing on self-knowledge, we can leverage historical data from multiple LLMs to build more robust and accurate correctness predictors. This is the core idea behind Generalized Correctness Models (GCMs). Unlike prior work that often restricted correctness models to the LLM generating the response, GCMs are trained on the historical correctness data from *many* different LLMs.
By systematically encoding this historical correctness information, GCMs learn generalizable patterns and strategies for predicting correctness that are applicable across various datasets and models. For instance, a GCM trained on data from eight different LLMs can learn to predict the correctness of an unseen model, even outperforming the self-emitted confidences of much larger LLMs that were not part of its training.
Key Factors Influencing Correctness Prediction
The research also delves into what truly drives correctness prediction. It identifies several crucial conditioning factors:
-
Answer Phrasing: How an LLM phrases and elaborates an answer is a strong predictor for correctness. The difference between a simple ‘4’ and ‘I believe the answer is 4’ can be significant.
-
World Knowledge: A Correctness Model’s ability to tap into its own parametric world knowledge to evaluate the likelihood of an answer being correct is a key enabler for generalization.
-
Question Type: Understanding that a model might perform differently on various types of questions (e.g., math vs. history) also contributes to accurate prediction.
Interestingly, the identity of the target model itself plays a less significant role, further supporting the model-agnostic nature of GCMs.
Alternative Methods for Injecting History
Beyond direct training, the study explores other ways to incorporate historical information:
-
In-Context Learning (ICL): Providing relevant prior examples directly within the prompt can improve correctness prediction, especially for larger models capable of leveraging verbalized reasoning.
-
Post-hoc Calibration: This method involves aligning a Correctness Model’s output confidences with historical ground truth. It’s particularly effective for adapting GCMs to new datasets with limited examples, significantly reducing calibration error.
Also Read:
- Improving AI Trust: A New Method to Calibrate Language Model Confidence
- Fact Grounded Attention: A New Approach to Reliable LLMs
Practical Benefits and Downstream Applications
The advantages of GCMs extend to practical applications. In tasks like selective prediction, where a system must decide whether to provide an answer or abstain based on its confidence, GCMs demonstrate superior performance. They achieve lower risk (fewer incorrect answers) at higher coverage (more answers provided) compared to traditional methods or even larger models’ self-reported confidences. This makes GCMs better suited for robust deployment in real-world scenarios.
In conclusion, the insight that LLMs lack special self-knowledge for correctness prediction is not a limitation but an opportunity. By training Generalized Correctness Models on the historical patterns of many LLMs, we can achieve highly accurate, calibrated, and model-agnostic confidence estimates, paving the way for more trustworthy and reliable AI systems.
