TLDR: This research explores improving the generalization of LLM uncertainty estimators by combining internal ‘hidden state’ features with ‘data-agnostic’ features. While data-agnostic features generally enhance cross-task performance, their impact is inconsistent, often due to the probe under-weighting them. The study suggests that pruning task-specific hidden units can amplify the importance of data-agnostic features, paving the way for more robust confidence estimation.
Large Language Models (LLMs) have become incredibly powerful, but they often generate responses that sound convincing even when they are factually incorrect. This “confident misinformation” can be a serious problem for users, especially in critical areas like healthcare. To make LLMs more reliable, it’s essential for them to not only provide answers but also to accurately estimate how confident they are in those answers.
One promising approach to gauge an LLM’s certainty involves looking at its “hidden states” – the internal representations generated as the model processes information. Researchers train external tools, called “probes,” on these hidden states to predict the factual accuracy of the LLM’s output. These internal signals are believed to hold valuable clues about the truthfulness of a response.
However, a significant challenge with this method is its limited ability to generalize. A probe trained to assess uncertainty for one type of task or dataset often struggles when applied to a different one. This suggests that the hidden states might be capturing information specific to the training task rather than universal indicators of truthfulness.
To overcome this limitation, a recent research paper, “Toward Better Generalisation in Uncertainty Estimators: Leveraging Data-Agnostic Features” by Thuy An Ha and Bao Quoc Vo, explores a novel solution. They propose combining these task-specific hidden-state features with “data-agnostic features.” Data-agnostic features are, as the name suggests, independent of any particular task or domain, making them potentially valuable for improving generalization.
The researchers conducted experiments using two popular LLMs, Llama2-7B and Mistral-7B, across various tasks including Factual Question Answering, Reading Comprehension, and Commonsense Reasoning. They investigated two key questions: first, whether adding data-agnostic features could improve the probe’s ability to generalize to new tasks or datasets; and second, if selecting only the most informative hidden-state features would further amplify this generalization by reducing task-specific noise.
The findings were insightful but not entirely straightforward. While incorporating data-agnostic features generally enhanced generalization performance in most scenarios, there were instances where their inclusion actually led to a decrease in accuracy. A similar mixed pattern emerged when only the most important hidden-state features were used; adding data-agnostic features didn’t consistently provide additional benefits compared to using the full set of hidden-state features.
A deeper analysis using SHAP values, which measure the contribution of each feature to the model’s predictions, shed some light on these inconsistencies. In successful cases, data-agnostic features consistently ranked higher in importance than the selected hidden-state features, amplifying their impact. However, in cases where performance degraded or didn’t improve as expected, the data-agnostic features were often underweighted by the trained probe, or intermixed with less important hidden-state features, thus failing to exert their full potential.
Also Read:
- Unlocking Smarter AI: How Large Language Models Are Learning to Reason on a Budget
- Unpacking LLM Confidence: Why AI Models Can Be Stubborn Yet Easily Swayed
This research highlights the complex interplay between task-specific and task-independent features in building robust uncertainty estimators for LLMs. While data-agnostic features show promise in improving generalization, their effective integration requires careful consideration of how the model prioritizes and utilizes them. The paper’s findings pave the way for future work aimed at developing more reliable and universally applicable methods for LLMs to express their confidence. For a comprehensive understanding of their methodology and results, you can find the full paper here.
