TLDR: This research paper introduces two novel algorithms, Optimal Weight (OW) and Inverse Surprising Popularity (ISP), for aggregating responses from multiple large language models (LLMs). OW leverages individual LLM accuracies (first-order information) to assign optimal weights, while ISP utilizes correlations between LLM answers (second-order information) to make better decisions, especially when ground-truth labels are unavailable. Both methods are theoretically proven and empirically shown to consistently outperform traditional majority voting across synthetic, benchmark, and real-world datasets, demonstrating a more robust and accurate approach to multi-agent LLM reasoning.
In the rapidly evolving landscape of large language models (LLMs), a significant challenge has emerged: how to effectively combine the answers from multiple LLMs to arrive at a single, more reliable conclusion. Traditionally, a simple approach called majority voting has been used, where the answer chosen by the most LLMs is selected. However, new research from MIT, Harvard University, The University of Chicago, and other institutions suggests that this method, while intuitive, overlooks crucial information about how LLMs perform.
A recent paper, “Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Information”, introduces two innovative aggregation algorithms designed to overcome the limitations of majority voting. These new methods, called Optimal Weight (OW) and Inverse Surprising Popularity (ISP), leverage what the researchers call “higher-order information” – essentially, deeper insights into the LLMs’ individual strengths and how their predictions relate to each other.
The Problem with Simple Majority Voting
Majority voting is a “zero-order” aggregation method. It treats every LLM’s answer equally, regardless of its typical accuracy or how often it agrees or disagrees with other models. This can be problematic because LLMs are not all the same; they have varying levels of accuracy and can exhibit correlations in their errors. Ignoring this “latent heterogeneity and correlation” means we might be missing opportunities to make more informed collective decisions.
Optimal Weight (OW): Leveraging Individual Strengths
The first new algorithm, Optimal Weight (OW), addresses this by incorporating “first-order information” – specifically, each LLM’s expected accuracy. Imagine you have a team of experts; you wouldn’t give equal weight to everyone if you knew some were consistently more accurate than others. OW does precisely this for LLMs. It assigns a unique weight to each LLM based on its accuracy, giving more influence to models that are typically more correct. The researchers theoretically prove that this weighted approach is “Bayesian-optimal,” meaning it maximizes the expected accuracy among all possible aggregation methods, given knowledge of the LLMs’ accuracies. It’s also shown to be more accurate than relying on any single LLM, under reasonable conditions.
However, a practical hurdle for OW is that knowing the true accuracy of an LLM often requires comparing its answers against many ground-truth labels, which can be very costly or even impossible in real-world scenarios.
Inverse Surprising Popularity (ISP): Learning from Correlations Without Ground Truth
This is where the second algorithm, Inverse Surprising Popularity (ISP), comes into play. ISP tackles the challenge of not having ground-truth labels by using “second-order information” – the correlations between different LLMs’ answers. Instead of needing to know the correct answer, ISP analyzes how often LLMs agree or disagree with each other on a given question. This information can be gathered by simply querying LLMs on many questions, a much less expensive process than human annotation.
ISP builds upon a concept called “surprising popularity” (SP), which was originally developed for human crowds. SP identifies answers that are chosen more often than expected, given what people predict others will say. However, the researchers found that the original SP method didn’t perform better than majority voting for LLMs because LLMs don’t exhibit the same systematic biases as human crowds. ISP cleverly modifies this by considering what predictions would be made if agents reported *other* answers, effectively amplifying prediction biases in a controlled way to better identify the correct answer. The theoretical analysis demonstrates that ISP consistently outperforms both majority voting and the original surprising popularity method.
Real-World Validation and Practical Impact
The researchers rigorously tested OW and ISP on various datasets, including simulated environments, popular LLM fine-tuning benchmarks like UltraFeedback and MMLU, and a real-world healthcare dataset from ARMMAN, a non-profit organization. Since true LLM accuracies are often unavailable in practice, they developed two heuristic approaches (OW-L and OW-I) to estimate these accuracies using the more readily available second-order information.
Across all experiments, the new methods consistently outperformed majority voting. For instance, on the UltraFeedback, MMLU, and ARMMAN datasets, where LLMs provided different answers for a significant portion of questions, OW and ISP yielded absolute accuracy gains ranging from 1.16% to 3.36% over majority voting. In many cases, these aggregation methods even surpassed the performance of the single best-performing LLM in the ensemble, demonstrating their ability to extend the capabilities of individual models.
Robustness Beyond Ideal Conditions
An important aspect of this research is its robustness. The theoretical findings hold even when the common assumption of “conditional independence” among LLM agents is relaxed. This means the methods are effective even if LLM predictions are correlated due to factors like varying question difficulty – for example, all LLMs performing well on easy questions and poorly on difficult ones. This makes the algorithms highly applicable to diverse real-world scenarios.
Also Read:
- Coordinating AI Agents: A Deep Dive into Reasoning-Aware Prompt Orchestration
- Interactive Learning: How LLMs Can Enhance Reasoning Through Peer Interaction
Looking Ahead
This work provides a significant step forward in multi-agent LLM reasoning, offering principled ways to combine LLM responses for more reliable outcomes without needing costly human labels. Future research could explore how to assign even more fine-grained, context-specific weights, incorporate even higher-order information, or integrate these aggregation frameworks with LLM post-training techniques to further enhance LLM capabilities.
