TLDR: A new multi-agent LLM framework improves judgmental forecasting by combining diverse perspectives and external evidence through structured argumentation (QBAFs). It uses ArgLLM, RAG-ArgLLM, and RbAM agents, merging their outputs to create more accurate and explainable predictions, especially with three agents and average score aggregation.
Predicting future events, a task known as judgmental forecasting, is crucial in many fields. However, relying solely on human judgment or even advanced Large Language Models (LLMs) can be challenging. LLMs, despite their capabilities, sometimes produce inaccurate or inconsistent information and lack transparent reasoning. A new research paper explores a novel approach to enhance these predictions by combining the strengths of multiple LLM agents through structured argumentation.
A Collaborative Approach to Forecasting
The paper, titled “Retrieval and Argumentation Enhanced Multi-Agent LLMs for Judgmental Forecasting,” introduces a multi-agent framework designed to improve claim verification. This framework treats future outcomes as claims that need to be assessed for plausibility. Instead of a single LLM making a prediction, several LLM-based agents work together, each contributing evidence and arguments for and against a claim.
These arguments are structured into what the researchers call Quantitative Bipolar Argumentation Frameworks (QBAFs). Imagine a structured debate where each point for or against a claim is an “argument,” and these arguments can attack or support each other. Each argument also has a “base score” indicating its initial strength.
Diverse Agents for Robust Reasoning
The framework employs three distinct types of LLM agents:
-
ArgLLM agents: These are baseline agents that generate and evaluate QBAFs using their inherent knowledge. They represent the LLM’s internal reasoning.
-
RbAM agents: Standing for Relation-based Argument Mining, these agents directly use external textual evidence as arguments. They classify whether retrieved information supports or attacks a claim, forming a QBAF from these external snippets.
-
RAG-ArgLLM agents: These agents combine the argumentative capabilities of ArgLLMs with Retrieval-Augmented Generation (RAG). They actively search for and incorporate external information to generate more factually grounded and comprehensive arguments, reducing the risk of LLM “hallucinations.”
Merging Perspectives for Better Outcomes
A core component of this framework is the “Multi-Agent QBAF Combinator.” This module acts like a mediator, taking the individual QBAFs generated by each agent and merging them into a single, more robust QBAF. It identifies arguments that are semantically similar across different agents and combines them. The strengths of these combined arguments are then aggregated, typically by averaging their base scores. This ensures that a diverse range of perspectives and evidence is integrated into a unified, transparent decision-making process.
Also Read:
- LLMs Learn to Self-Improve Through Multi-Agent Co-Evolution
- Enhancing Factual Accuracy in Long-Form AI-Generated Content with Multi-Agent Debates
Experimental Insights
The researchers, Deniz Gorur, Antonio Rago, and Francesca Toni, tested their framework on two judgmental forecasting datasets: GJOpen and Metaculus. They experimented with various combinations of two and three agents, utilizing different base LLMs like Mixtral, Mistral, Gemma, Llama-3, and GPT-4o. External sources for the RAG and RbAM agents included news abstracts from the NYTimes and The Guardian.
Key findings from their experiments include:
-
RAG-ArgLLM agents consistently outperformed standard ArgLLM agents, especially on the Metaculus dataset, underscoring the value of external knowledge.
-
Combining QBAFs from multiple agents often led to improved forecasting accuracy, particularly when the agents brought complementary argumentative reasoning to the table.
-
The three-agent combination (one ArgLLM and two RAG-ArgLLMs with different external sources) showed further enhancements in accuracy, demonstrating that greater diversity in reasoning and information sources can lead to more reliable predictions.
-
Using an average to aggregate argument base scores generally yielded better results than simply taking the maximum score, suggesting a balanced approach is more effective.
This research highlights a promising direction for improving judgmental forecasting by leveraging multi-agent LLM systems. By structuring debates, incorporating external evidence, and combining diverse perspectives, the framework offers a more explainable and accurate method for predicting future events. For a deeper dive into the methodology and detailed results, you can find the full paper here.
