TLDR: This paper introduces “argumentative coherence” for judgmental forecasting, a property ensuring a forecaster’s reasoning aligns with their prediction. Evaluations show that filtering incoherent predictions improves accuracy for both human and AI forecasters. However, a crowd-sourced study reveals that users don’t naturally exhibit this coherence, suggesting the need for built-in mechanisms to enforce it in forecasting systems.
Forecasting the future is a complex endeavor, often relying on human judgment rather than just historical data. This approach, known as judgmental forecasting, is crucial in many fields, but it’s also susceptible to human biases and irrationality, which can lead to inaccurate predictions. A new research paper, “Argumentatively Coherent Judgmental Forecasting,” explores a novel way to improve these predictions by introducing a concept called argumentative coherence.
The core idea behind argumentative coherence is simple yet powerful: a forecaster’s reasoning should align logically with their final prediction. If someone believes certain arguments strongly support a particular outcome, their prediction should reflect that conviction. Conversely, if their reasoning points away from an outcome, their prediction should follow suit. This paper formally defines this property, making it adaptable to various forecasting scenarios through adjustable parameters.
To test the practical value of this coherence, the researchers conducted three distinct evaluations. The first involved an in-person study using a system called ArguCast, which facilitates argumentative debates around forecasting questions. Participants used ArguCast to generate debates, express opinions, and make predictions on real-world events like sports outcomes, political elections, and company profitability. When the predictions from incoherent forecasters were filtered out, the aggregated forecasts consistently moved closer to established baselines from prediction markets and expert systems. This suggests that enforcing coherence can indeed lead to more accurate group predictions.
The second evaluation focused on Large Language Models (LLMs), which have recently shown impressive capabilities in forecasting. The researchers used Argumentative LLMs (ArgLLMs) to generate supporting and attacking arguments for forecasting questions, effectively creating argumentative structures similar to those in ArguCast. When coherence was applied to these LLM-generated forecasts, accuracy improved across different models, including Mistral, Mixtral, Llama 3, and GPT-4o. This highlights that the principle of coherence is beneficial not only for human forecasters but also for AI-driven prediction systems.
The final evaluation was a crowd-sourced user study designed to understand how naturally users align with the proposed notion of coherence. Participants were presented with various debate scenarios and asked to assess a fictitious forecaster’s prediction. Surprisingly, the results indicated that users do not inherently align with the researchers’ definition of coherence. This finding is significant because it underscores the need for built-in mechanisms within judgmental forecasting systems to identify and filter out incoherent opinions before aggregating group predictions. Despite users not being naturally coherent, the study also found that their alignment with the coherence notion improved when the debate scenarios were more complex, suggesting that increased cognitive effort in reasoning might help overcome biases.
Also Read:
- AI’s Self-Awareness: A New Metric for Effective Human-AI Collaboration
- Transparent AI Collaboration: SynLang and the Philosophy of Shared Intelligence
In conclusion, the research demonstrates that argumentative coherence is a valuable property for improving judgmental forecasting accuracy, whether performed by humans or advanced AI models. While its benefits are clear, the study also reveals that human users may require explicit guidance or automated filtering to ensure their predictions are argumentatively coherent. This work paves the way for more robust and reliable forecasting systems that integrate structured reasoning and coherence constraints. You can read the full research paper here: Argumentatively Coherent Judgmental Forecasting.
