TLDR: Reasoning in large language models (LLMs) improves overall accuracy but can significantly reduce recall at critical low false positive rates (FPRs) in precision-sensitive tasks like safety and hallucination detection. This is because reasoning makes LLMs overconfident in their errors. “Think Off” (no reasoning) performs better at low FPRs, while a simple ensemble of “Think On” and “Think Off” modes can combine the benefits of both.
The world of large language models (LLMs) is constantly evolving, with “reasoning” becoming a key feature to boost their performance. While reasoning often improves overall accuracy across various tasks, a new study titled “Reasoning’s Razor: Reasoning Improves Accuracy but Can Hurt Recall at Critical Operating Points in Safety and Hallucination Detection” by researchers from Apple, including Atoosa Chegini and Hamid Kazemi, reveals a crucial trade-off. This research highlights that while reasoning-augmented LLMs are generally more accurate, they can surprisingly underperform in situations where precision is paramount, such as detecting unsafe content or hallucinations.
The study focuses on two critical classification tasks: safety detection (identifying unsafe text) and hallucination detection (spotting factually incorrect responses in Retrieval-Augmented Generation, or RAG, systems). These tasks demand extremely low false positive rates (FPRs), often below 1%, because false alarms can have severe consequences. For example, incorrectly flagging a safe user query as unsafe can block legitimate interactions, frustrating millions of users. Similarly, misidentifying a correct response as a hallucination can lead to unnecessary re-generations, adding latency and computational cost.
The researchers compared two inference modes: “Think On,” where LLMs generate intermediate reasoning steps before making a decision, and “Think Off,” where decisions are made directly without explicit reasoning. Their findings consistently show that Think On models achieve higher overall accuracy. However, when evaluated at very low FPR thresholds—the critical operating points for real-world deployment—Think Off models significantly outperform Think On models in terms of recall (True Positive Rate). For instance, a fine-tuned safety classifier showed a 2.9 times degradation in recall at 1% FPR when reasoning was enabled (13.8% recall) compared to when it was disabled (40.0% recall).
This unexpected drop in recall with reasoning is attributed to how reasoning affects model confidence. The study found that Think On models tend to produce highly polarized confidence scores, meaning predictions are often made with near-certainty, even when they are incorrect. This “overconfidence in errors” makes it difficult to distinguish between correct and incorrect predictions under strict FPR constraints. The confidence in an incorrect classification was observed to steadily increase as the reasoning chain unfolded. In contrast, Think Off models exhibited more moderate confidence distributions, allowing for better separation of correct and incorrect predictions at stringent thresholds.
The research also explored different methods for confidence scoring. Token-based scoring, which derives confidence from the probability of class tokens, was found to be substantially superior to “self-verbalized confidence,” where models explicitly state their uncertainty in natural language. Self-verbalized confidence often failed completely at low FPR thresholds, yielding zero recall for several datasets. Interestingly, reasoning had opposite effects on these two scoring methods: it degraded performance at low FPR for token-based scoring but showed modest improvements for self-verbalized scoring.
To address this trade-off, the researchers proposed a simple ensemble approach, combining the confidence scores from both Think On and Think Off modes with equal weighting. This ensemble successfully leveraged the strengths of both, achieving high overall accuracy while maintaining practical low-FPR recall. For example, in hallucination detection, the ensemble either matched or improved upon the best performance of individual modes.
Another mitigation strategy explored was “certainty-level prompting,” where models are instructed to classify as positive (e.g., “unsafe”) only if they are at least a certain percentage confident. While this improved Think On performance, especially for safety classification, it still couldn’t fully close the performance gap with Think Off at strict operating points.
Also Read:
- Bridging the Gap: How Symbolic AI Enhances Transparency and Reasoning in Large Language Models
- Unmasking Self-Jailbreak: A Framework for Safer Large Reasoning Models
In conclusion, the study positions reasoning as a “double-edged tool” for classification tasks. While beneficial for average accuracy, it can be ill-suited for applications requiring strict precision due to its tendency to inflate confidence in errors. The findings underscore the importance of carefully considering deployment requirements and choosing appropriate inference strategies, such as using Think Off for precision-sensitive tasks or employing ensemble methods, when deploying LLMs for safety-critical applications. For more details, you can refer to the full research paper available at arXiv:2510.21049.
