TLDR: ROUNDTABLEPOLICY is an innovative framework that enhances the scientific reasoning and narrative generation capabilities of large language models (LLMs). By mimicking scientific committees, it uses a confidence-weighted consensus from multiple LLMs (‘players’) evaluated by other LLMs (‘graders’). This process builds a ‘confidence-weight table’ that guides a ‘fusion agent’ to synthesize more accurate, coherent, and less hallucinated responses than individual models, significantly improving performance in both diverse scientific tasks and structured proposal writing.
Large language models (LLMs) have shown incredible potential in generating text and aiding scientific discovery. However, they often struggle with accuracy, sometimes making illogical jumps or producing fragmented explanations. This can be a significant problem in scientific fields where precision and clarity are paramount. Imagine a scientific proposal that needs to be factually correct and also structured persuasively; a single LLM might not always meet these diverse demands.
To address these challenges, researchers have introduced a new framework called ROUNDTABLEPOLICY. This approach is inspired by how scientific committees work and by the concept of ‘The Society of Mind,’ where many specialized agents coordinate to achieve intelligence. Instead of relying on one LLM, ROUNDTABLEPOLICY uses a weighted consensus from multiple LLMs to improve reasoning and create better scientific narratives.
Here’s how it works: Different LLMs, called ‘players,’ propose their answers to a given problem. These responses are then evaluated by an ‘AI Committee’ made up of other LLMs, known as ‘graders.’ The graders assess the quality of each player’s response based on specific criteria and also provide a confidence level for their assessment. Over time, this process builds a ‘confidence-weight table,’ which acts like a reputation system, tracking how reliable and uncertain each player is across different types of tasks or evaluation dimensions.
When it’s time to generate a final answer, a ‘fusion agent’ uses this confidence-weight table to intelligently combine the diverse responses from the players. It prioritizes contributions from models that have historically shown higher reliability and lower uncertainty. This method ensures that the final answer is not just an average, but a carefully constructed consensus that balances factual accuracy with narrative coherence, all while maintaining transparency in how the decision was reached. This is a key difference from other multi-LLM methods that might converge on an answer without clearly showing why.
Real-World Performance
The effectiveness of ROUNDTABLEPOLICY was tested in two main scenarios. In the ‘MultiTask’ setting, LLMs tackled a variety of complex scientific questions from fields like geoscience, biology, mathematics, and materials science. The results showed a significant improvement, with an average lift of 13.01% in factual reliability and reasoning quality compared to individual models. In the ‘SingleTask’ setting, models were asked to generate structured scientific proposals, including sections for Background, Methodology, and Impact. Here, ROUNDTABLEPOLICY achieved an 11.04% average improvement in narrative coherence, creativity, scientific rigor, and logical consistency.
Further analysis confirmed that these improvements were statistically robust and consistent over many evaluations. The framework proved particularly good at elevating the performance of weaker LLMs without diluting the strengths of stronger ones. The study also looked at the AI graders themselves, finding that they showed high agreement on objective tasks but, much like human reviewers, had more varied opinions on subjective aspects like creativity. However, by aggregating judgments from multiple AI graders, the system effectively mitigated individual biases.
Also Read:
- Guiding Language Models for Better Tool Use and Clearer Decisions
- Multi-Agent AI Framework Enhances Radiology Report Generation and Evaluation
Why Consensus Matters
An important finding from the research was that a weighted consensus approach consistently outperformed simply picking the best single model. This highlights that robust scientific progress often comes from collective deliberation and the synthesis of multiple viewpoints, rather than relying on one dominant voice. The study also revealed that the system’s ‘institutional memory’ (the confidence-weight table) needed more training rounds to become effective for diverse, heterogeneous tasks, but converged much faster for more structured writing tasks.
ROUNDTABLEPOLICY represents a significant step towards making AI-assisted scientific discovery more trustworthy and effective. By formalizing deliberation among LLMs into a principled process, it offers an interpretable and generalizable path for enhancing reasoning and communication in science. Future work aims to make the system even more adaptive, potentially by integrating reinforcement learning and richer feedback signals. You can learn more about this research in the full paper available at arXiv:2509.16839.
