TLDR: A new study reevaluates the effectiveness of self-consistency in modern large language models (LLMs) like Gemini 2.5. It finds that while sampling multiple reasoning paths improves accuracy, the gains plateau after a moderate number of samples, with computational costs increasing linearly. The research suggests that diminishing returns are due to redundancy in reasoning paths, advocating for optimized sampling strategies to balance accuracy and efficiency.
In the rapidly evolving landscape of large language models (LLMs), techniques to enhance reasoning capabilities are constantly being refined. One such method, self-consistency, has been a cornerstone in improving the reliability of LLM outputs by sampling multiple reasoning paths and selecting the most consistent answer. Originally, this approach showed significant gains, especially with earlier, smaller models, by mitigating stochastic errors and boosting robustness through aggregation.
Revisiting Self-Consistency in Modern LLMs
A recent study by Chiyan Loo, titled “Reevaluating Self-Consistency Scaling in Multi-Agent Systems,” delves into how these claims hold up with today’s advanced models, specifically Gemini 2.5 Flash Lite and Gemini 2.5 Pro. The research aimed to understand the trade-offs of increasing the number of sampled reasoning paths, questioning whether the same improvement curve persists as model design and training have advanced.
The study’s methodology involved a structured self-consistency framework. Multiple independent reasoning agents, all instantiated from the same LLM, generated separate chain-of-thought (CoT) responses for each query. A secondary aggregator model then analyzed these traces to determine the most internally consistent response. Agent counts of 3, 5, 10, 15, and 20 were tested, with a single-agent CoT baseline serving as the control. The experiments were conducted on two distinct datasets: HotpotQA, a multi-hop question answering dataset, and Math-500, a collection of mathematics problems. Performance was assessed based on accuracy and total token consumption, which reflects computational cost and latency.
Key Findings: Diminishing Returns and Efficiency
The results largely confirmed earlier findings: self-consistency does improve accuracy, but these gains diminish as the number of agents increases, eventually reaching a plateau. On HotpotQA, the Gemini 2.5 Flash Lite model showed an irregular but generally upward and plateau-like trend, with accuracy differences of roughly 0.4% at 20 agents compared to the baseline. Token usage, however, increased almost linearly with each additional agent, highlighting a significant rise in computational cost.
For the Math-500 dataset, Flash Lite’s accuracy improved steadily up to 10 agents before plateauing and even slightly declining past 15 agents. The Gemini 2.5 Pro model, tested on Math-500 with up to 15 sampled paths, exhibited a smoother and more consistent improvement curve. Its accuracy rose from 98% (CoT baseline) to 99.2% with 3 agents and 99.6% with 15 agents, a total improvement of 1.6%. This steady but limited rise further supports the plateau behavior observed in previous work.
These findings suggest that the diminishing returns are primarily driven by the overlap among reasoning paths rather than inherent differences in model capability. While larger models like Gemini 2.5 Pro showed more stable gains, even smaller models like Flash Lite followed the same general trend. The research concludes that while self-consistency remains a valuable technique, high-sample configurations offer little marginal benefit relative to their computational cost. A moderate number of sampled reasoning paths appears to strike the best balance between accuracy and efficiency.
Also Read:
- Unveiling the ‘Accuracy Cliff’: Why LLMs Struggle with Long Deterministic Tasks
- Adaptive Effort Control: AI Models Learn to Optimize Reasoning for Cost and Accuracy
Implications for Future LLM Applications
The study underscores the importance of optimizing sampling strategies in self-consistency. Developers and researchers should consider the trade-off between accuracy gains and computational expense. Techniques like adaptive-consistency, which dynamically adjust sample generation based on interim agreement, or reasoning-aware self-consistency, which uses criteria-based early stopping, could offer more efficient ways to leverage multi-path reasoning without incurring excessive costs.
For more details, you can read the full research paper here.
