TLDR: The UNH team participated in CheckThat! Task 2, researching methods for extracting verifiable claims from social media posts. They compared fine-tuning LLMs, like FLAN-T5 Large (which achieved the highest METEOR score), with various prompting techniques, including self-refinement and keyword-based few-shot prompting. The study found that while fine-tuning excelled in automated metrics, prompting methods often produced claims more useful for human fact-checkers, highlighting the limitations of current evaluation metrics and the complexities of real-world claim extraction.
In the rapidly evolving landscape of social media, the spread of misinformation and unchecked claims poses a significant challenge. To combat this, the CheckThat! Lab aims to develop computational tools that assist human fact-checkers. The University of New Hampshire (UNH) team participated in CheckThat! Task 2 (English) in 2025, focusing on a crucial aspect: extracting main, check-worthy claims from social media posts in a succinct and concise manner.
The research paper, titled “UNH at CheckThat! 2025: Fine-tuning Vs Prompting in Claim Extraction,” explores various methods, including different prompting techniques like few-shot prompting and fine-tuning with diverse Large Language Model (LLM) families. The goal was to understand the trade-offs between optimizing automated evaluation scores, such as METEOR, and generating claims that are genuinely useful for human fact-checkers.
Challenges in Claim Extraction
The UNH team identified several unique challenges in this task. Often, social media posts contain multiple relevant claims, but the task requires focusing on a single, verifiable fact. Another issue was the presence of missing multi-modal content; many original posts included photos or videos that were not available in the dataset, leading models to make educated guesses. A significant concern was also the problem of ‘faithful extractions,’ where LLM hallucinations could introduce content not present in the original post, making the extracted claim overly verbose or inaccurate.
Approaches Explored
The team cast a wide net, categorizing their approaches into three main areas: fine-tuning, prompting, and “frustratingly easy” baselines. They utilized resources provided by the task organizers and publicly available LLMs from Hugging Face and the Together.AI API service.
Fine-Tuning Approaches
The most successful method in terms of the METEOR score was **Finetuned Flan-T5 Large**. This model, with 783 million parameters, was fine-tuned on the CLEF 2025 Task 2 training dataset. Despite its smaller size compared to billion-parameter models, it offered the best compromise between capability and practical feasibility under resource constraints. The training involved a straightforward task-specific prompt and ran for 10 epochs on an NVIDIA 4060 GPU, achieving an average validation-set METEOR score of 0.5569.
Another fine-tuning method involved **LoRA fine-tuning of Flan-T5 Base**. LoRA (Low-Rank Adaptation) is an efficient technique that updates only a small percentage of model parameters, balancing performance with computational efficiency. This approach yielded a validation METEOR score of 0.3054.
The team also fine-tuned the **DeepSeek-R1-Distill-Llama-8b** causal language model, which achieved a validation METEOR score of 0.2541. The researchers noted that achieving significant improvements with moderate-sized LLMs through fine-tuning, even with LoRA, often required computational resources beyond their budget.
Prompting Approaches
Prompting-based methods explored variations of few-shot prompting, self-refinement, self-scoring, and post-processing. One notable method was **Claimifying Social Media Posts with Self-Refinement**. This approach used an iterative self-refinement stage where the LLM generated an initial claim, provided feedback on its check-worthiness, and then refined the claim based on that feedback. Models like GPT-4.1-nano, Gemini-2.0-Flash, LLaMA-3.3-70B, and Grok3 were explored, with Grok3 combined with few-shot Chain-of-Thought prompting achieving the highest average METEOR score of 0.332 in this category.
The **Keyword Few-Shot (KBFP) and Self-Refine** method focused on intelligently selecting few-shot examples using keyword matching from the target social media post. While KBFP alone yielded a higher METEOR score, adding one iteration of Self-Refine often produced more concise and less redundant claims, even if the METEOR score was slightly lower. Repeated applications of Self-Refine, however, tended to introduce verbosity and hallucinations.
A **Subclaim Extraction and Filtering with Refinement** approach involved extracting multiple potential “subclaims,” scoring and filtering them, and then synthesizing a final main claim. This multi-stage pipeline aimed for a balance of comprehensiveness and clarity.
The **Max Multi-Prompt** approach addressed nuances like references to images or sarcastic tones in social media posts by designing targeted prompts. For instance, a prompt might instruct the model to imagine searching for a referenced image and describe its likely content. This method demonstrated the potential benefits of triaging between multiple prompt templates.
Baseline Approach
A “frustratingly easy” **Regurgitation Baseline** was designed, which simply reused the original social media post or a truncated version as the extracted claim. Surprisingly, truncating the post to 100 characters yielded a METEOR score of 0.2429, setting a strong reference point.
Also Read:
- Unpacking LLM Effectiveness: Fine-Tuning Outperforms In-Context Learning for Misinformation Detection
- Assessing Topic Model Quality with Large Language Models: A New Framework
Evaluation and Key Findings
All approaches were evaluated on a validation subset using the METEOR metric. While fine-tuning Flan-T5-Large achieved the highest METEOR score of 0.5569, the researchers noted that higher-quality and more useful claims for human fact-checkers were sometimes extracted using other methods, even with lower METEOR scores. For example, Self-Refine combined with KBFP or Claimify produced claims that subjectively better highlighted the core assertion, such as the example of Joe Biden’s house appearing too expensive for his reported salary.
The study highlighted that the METEOR metric, while useful, is limited to overlap with gold-standard claims and doesn’t fully capture the utility for human fact-checkers. Issues with the gold claims themselves were also observed, as they sometimes omitted critical details or referenced unavailable multi-modal content.
Overall, the UNH team’s participation in CheckThat! Task 2 provided valuable insights into the strengths and weaknesses of different LLM approaches for claim extraction. While fine-tuning can optimize for specific metrics, prompting strategies, especially with iterative refinement, show promise in generating more actionable and contextually rich claims for real-world fact-checking scenarios. You can read the full research paper here: UNH at CheckThat! 2025: Fine-tuning Vs Prompting in Claim Extraction.
