TLDR: The AIRwaves team developed a two-stage information retrieval pipeline to link implicit scientific claims on social media to their original publications. This system, which ranked second in the CLEF 2025 CheckThat! Lab, combines a fine-tuned E5-large dual encoder with a SciBERT cross-encoder re-ranker. Key findings include significant performance gains from incorporating hard negative training examples and the crucial role of neural re-ranking in boosting retrieval accuracy, achieving an MRR@5 of 0.6828 and placing the correct document first for 62% of queries.
In the rapidly evolving landscape of social media, scientific claims often circulate without direct links to their original research. This presents a significant challenge for fact-checkers and those seeking evidence-based information. A recent study, presented at the CLEF 2025 CheckThat! Lab, tackles this problem head-on by developing a sophisticated system to connect implicit scientific claims on social media to their corresponding academic papers.
The research, conducted by the AIRwaves team, introduces a two-stage information retrieval pipeline designed to overcome the inherent difficulties of this task. These challenges include the brevity and informal nature of social media posts (queries), the specialized language used in scientific documents, and the ‘lexical sparsity’ – meaning tweets often paraphrase findings without using exact keywords from the papers.
The Two-Stage Retrieval System
The core of the AIRwaves approach lies in its two-stage pipeline. The first stage, known as Neural Representation Learning, employs a ‘dual encoder’ model. This model, specifically an E5-large variant, is fine-tuned to create numerical representations (embeddings) for both social media claims and scientific documents. These embeddings allow the system to understand the semantic meaning of the text, rather than just relying on keyword matching. A key innovation in this stage was the use of ‘in-batch’ and ‘mined hard negatives’ during training. Hard negatives are documents that are similar to the correct one but ultimately incorrect, forcing the model to learn finer distinctions. The team also enhanced this stage with ‘chunked tokenization’ and rich document metadata, such as titles, abstracts, authors, journals, and sources.
The second stage is Neural Re-Ranking. After the dual encoder generates an initial list of candidate documents, a ‘cross-encoder’ model refines this list. Unlike dual encoders, cross-encoders process the query and each candidate document together, allowing for a deeper, more contextual understanding of their relationship. The SciBERT model, a BERT variant pre-trained on scientific texts, proved to be the most effective re-ranker in this stage.
Key Findings and Performance
The AIRwaves team’s system achieved remarkable results, securing second place in Subtask 4b of the CLEF 2025 CheckThat! Lab. The optimized sparse-retrieval baseline (BM25) achieved an MRR@5 (Mean Reciprocal Rank at 5) of 0.5025. By replacing purely lexical matching with neural representations, performance significantly improved to MRR@5 = 0.6174. The complete two-stage pipeline further boosted this to an impressive MRR@5 = 0.6828.
Several important insights emerged from their experiments:
- Hard Negatives Matter: Incorporating BM25-mined hard negative training examples significantly improved the query-to-document retrieval performance. This technique forces the model to learn more subtle semantic patterns.
- Document Metadata: While title and abstract were the primary drivers of retrieval performance, adding author information provided the most significant standalone gain among other metadata fields like journal and source. However, the overall improvement from adding all metadata fields was modest and not statistically significant.
- Re-Ranking is Crucial: The neural re-ranking stage, particularly with SciBERT, delivered a substantial and statistically significant boost in retrieval quality. It improved the MRR@5 by 0.07 absolute points over an already strong dense retrieval model. The system was able to place the correct document in the first position for approximately 62% of queries, a significant improvement over the baseline.
Also Read:
- Enhancing Language Models with Specialized Knowledge through Iterative Reinforcement Learning
- Disentangling Information for Smarter Recommendations
Implications for Fact-Checking and Research
The findings demonstrate that combining dense retrieval with neural re-rankers offers a powerful and efficient solution for matching social media claims to scientific studies. For real-world applications, the system can immediately match roughly 62.8% of posts to the correct publication. If a user is willing to manually review a few documents, the correct paper will be found within the top 10 suggestions approximately 79.5% of the time. This significantly narrows down the search space for fact-checkers and researchers, making evidence-based discourse more accessible.
Future work could explore dynamic hard negative mining and the use of generative models like MonoT5 for re-ranking. This research provides a practical blueprint for future evidence-retrieval pipelines, enhancing our ability to navigate the complex interplay between social media and scientific information. You can read the full research paper here.
