TLDR: This research introduces FISCORE, a novel reinforcement learning framework designed to enhance the reliability of Large Language Models (LLMs) by teaching them to abstain from answering questions beyond their knowledge scope. Unlike previous methods that use coarse-grained signals, FISCORE employs a fine-grained semantic confidence reward. It works by sampling multiple answers, semantically clustering them, and then training the LLM to retain answers from high-confidence clusters while discarding those from low-confidence ones. The paper also proposes a new reliability metric that balances helpfulness and truthfulness, demonstrating FISCORE’s superior performance and generalization across various datasets and LLMs.
Large Language Models (LLMs) have become incredibly powerful tools, capable of generating content, performing complex reasoning, and even assisting in software development. However, a significant challenge remains: their tendency to “hallucinate,” or fabricate plausible but false information when asked questions outside their knowledge. This issue can severely undermine user trust and lead to the spread of misinformation, especially in critical fields like medicine or law.
To address this, researchers are exploring ways to teach LLMs to “abstain” from answering questions they don’t know, rather than guessing. Existing methods often rely on broad, overall confidence scores or uncertainty measures. The problem with these approaches is that they might not give the LLM a precise understanding of its own knowledge boundaries, leading to either too much abstention (refusing to answer questions it actually knows) or too little (answering confidently but incorrectly).
Introducing FISCORE: Fine-grained Semantic Confidence Reward
A new research paper, titled “TEACHINGLLMS TOABSTAIN VIAFINE-GRAINED SEMANTICCONFIDENCEREWARD” by Hao An and Yang Xu, proposes a novel solution called FISCORE. This framework uses reinforcement learning to guide LLMs to abstain based on a more detailed, “fine-grained” understanding of confidence for each specific answer. You can read the full paper here.
Instead of a single, overall confidence score, FISCORE works by having the LLM generate multiple possible answers to a question. These answers are then grouped into “semantic clusters” based on how similar their meanings are. The size of a cluster acts as a proxy for confidence: a large cluster means many generated answers agree, suggesting high confidence, while small or single-answer clusters indicate low confidence or disagreement.
The LLM is then trained to align its expressed confidence (e.g., saying “sure” or “unsure”) with this intrinsic semantic confidence. It receives a positive reward when it correctly identifies high-confidence answers to keep and low-confidence answers to discard. This teaches the model a more nuanced and accurate way to decide when to answer and when to abstain.
A New Way to Measure Reliability
The paper also highlights a common challenge: improving an LLM’s “truthfulness” (abstaining from false information) often comes at the cost of its “helpfulness” (answering correctly when it knows). To better evaluate this balance, the authors propose a new metric called F1rel. This metric combines helpfulness and truthfulness into a single, comprehensive score, providing a more accurate assessment of an LLM’s awareness of its own knowledge limits.
How FISCORE is Trained
FISCORE leverages a reinforcement learning algorithm called Group Relative Policy Optimization (GRPO), which is well-suited for handling multiple generated responses per question. The LLM is prompted to generate an answer within <answer> tags and then state its confidence (“sure” or “unsure”) within <confidence> tags. This structured output helps the model reflect on its confidence after generating an answer, with “unsure” being interpreted as an act of abstention.
The total reward during training is a combination of three components: the fine-grained semantic confidence reward (encouraging alignment with semantic consensus), an accuracy reward (preventing the model from confidently giving incorrect but consistent answers), and a format reward (ensuring the model follows the specified output structure).
Also Read:
- Adaptive Prediction for Self-Evolving AI: Quantifying LLM Confidence
- Unlocking Smarter LLM Reasoning: How Internal Confidence Guides Learning
Experimental Results and Insights
The researchers tested FISCORE on various question-answering datasets, including both in-domain (Pararel) and out-of-distribution (TriviaQA, Natural Questions, SciQ) benchmarks, using models like Llama3-8B-Instruct and Qwen2.5-7B-Instruct. The results show that FISCORE significantly improves overall reliability (F1rel) compared to other methods, especially on out-of-distribution data. This suggests that the knowledge boundaries learned by FISCORE are more robust and generalizable.
A key finding was that the fine-grained semantic confidence reward is more effective than coarser, entropy-based uncertainty signals used in other reinforcement learning approaches. Interestingly, the study also revealed that a model’s raw accuracy or extensive knowledge capacity doesn’t automatically translate to precise self-awareness of its knowledge boundaries, underscoring the importance of dedicated abstention training.
While FISCORE demonstrates strong performance, the authors acknowledge some limitations, such as the need to manually set a threshold for abstention and the increased computational cost due to generating multiple answers and confidence expressions. Nevertheless, this research offers a promising direction for making LLMs more reliable and trustworthy by teaching them to intelligently recognize and communicate their knowledge gaps.
