TLDR: Researchers developed a “council debate” method where multiple large language models (LLMs) collaborate to detect scientific claims, references, and entities in tweets. While not top-ranked for claims or entities, their method achieved first place in identifying references to scientific studies, demonstrating that multi-LLM debates outperform individual models for this task.
In the rapidly evolving digital landscape, distinguishing scientific information from general online chatter is crucial. Researchers from TOBB University of Economics and Technology and Qatar University have introduced an innovative approach to tackle this challenge, particularly for content found on social media platforms like Twitter. Their work, presented at CheckThat! 2025, focuses on using large language models (LLMs) in a unique “council debate” framework to identify scientific discourse.
The core of their research, detailed in their paper “TurQUaz at CheckThat! 2025: Debating Large Language Models for Scientific Web Discourse Detection”, revolves around three specific categories for classifying tweets: whether they contain a scientific claim, if they reference a scientific study or publication, or if they mention scientific entities such as universities or scientists.
Simulating Academic Discussions with LLMs
The team explored three distinct debating methods for LLMs: single debate, team debate, and council debate. In the single debate setup, two LLMs argue opposing viewpoints while a third acts as a judge. The team debate expands on this, with multiple models collaborating within each side before presenting their collective arguments. The most promising method, and the one chosen as their primary model, is the council debate. Here, multiple expert LLMs deliberate together to reach a consensus, guided by a chairperson model. This collaborative approach aims to reduce individual model biases and foster more reliable decision-making.
The researchers utilized a variety of LLMs in their experiments, including open-source models like Gemma3, Qwen3, DeepSeek-R1, Phi4, Mistral, and LLaMA 3.1, as well as commercial models such as o4-mini and Claude-4. The council debate configuration involved five council members and Llama3.1 as the chairperson, demonstrating a diverse panel of AI experts working together.
Performance and Key Findings
The effectiveness of these debating methods was evaluated on the CheckThat! 2025 Task 4a datasets. While their proposed method, specifically the council debate, did not achieve top rankings for identifying scientific claims (8th out of 10) or mentions of scientific entities (9th out of 10), it remarkably secured the first place in detecting references to scientific studies or publications. This highlights a particular strength of their debate-based LLM approach in pinpointing direct links to academic work.
A significant observation from their experiments was that all debate-based approaches outperformed individual LLMs used as baselines. This suggests that the structured interaction and collective reasoning among multiple LLMs lead to improved performance in scientific discourse detection compared to a single model making a prediction. The team debate method also showed improvements over the single debate, especially with different team configurations.
Also Read:
- Evaluating Rerankers for Scientific AI: Insights from the New SciRerankBench
- Enhancing Portuguese Fact-Checking: A Semi-Automated Approach to Enriching News Datasets
Future Directions
The authors plan to extend their debating framework to other classification tasks and further investigate the impact of prompt design and the integration of additional LLMs. This research paves the way for more sophisticated AI systems capable of navigating and categorizing complex information in online environments, contributing to better information hygiene in the scientific web discourse.
