TLDR: This research develops a semi-automated method to enrich Portuguese fake news datasets with external evidence. It uses Large Language Models (Gemini 1.5 Flash) to extract claims and Google search APIs to retrieve supporting or refuting documents. The study validates this approach, showing how data cleaning and external context improve fake news detection model performance, highlighting the importance of external evidence for robust fact-checking in Portuguese.
In today’s digital age, especially since 2015, the way we consume information has drastically changed. Social media has become a primary source of news for many, particularly younger generations. However, this rapid spread of information has also led to a significant increase in misinformation and fake news, which can cause considerable harm to individuals, companies, and governments.
Manual fact-checking organizations, like Agência Lupa and Boatos.org in Brazil, play a vital role in verifying information. Yet, the sheer volume and speed at which misinformation spreads far exceed the capacity of human fact-checkers. This challenge became even more apparent during the COVID-19 pandemic, when internet usage surged, and questionable content circulated widely.
This limitation of manual verification has driven the development of automated or semi-automated fact-checking (SAFC) tools. These systems aim to analyze the truthfulness of claims by comparing them with external knowledge sources, integrating techniques from Information Retrieval (IR) and Natural Language Processing (NLP). While full automation still faces challenges, many systems operate semi-automatically, where technology assists human experts in the final validation.
In the context of the Portuguese language, despite existing research and resources for fake news detection, there’s a notable gap. Most publicly available datasets focus on classifying news based on intrinsic text features, like writing style, rather than providing external evidence. This lack of external evidence is crucial for developing robust SAFC systems that verify facts against external sources.
A recent dissertation by Juliana Resplande Sant’Anna Gomes addresses this very gap. The core objective of this work is to develop a method to enrich existing Portuguese news datasets by adding relevant contextual evidence retrieved from external sources. To achieve this, three prominent datasets with diverse characteristics were selected: Fake.Br (general web news), COVID19.BR (WhatsApp messages about health), and MuMiN-PT (general tweets).
The proposed approach simulates how a user might seek additional information to verify a news story. It uses Large Language Models (LLMs), specifically Gemini 1.5 Flash, to extract the main claim from the original text, especially when a direct search doesn’t yield strong matches. This extracted claim then serves as an optimized query for search engine APIs, such as the Google Search API and the Google FactCheck Claims Search API, to retrieve relevant external documents, which are then linked to the original dataset item.
Ensuring Data Quality
Before enrichment, the datasets underwent a rigorous semi-automatic validation process. This involved automated routines combined with detailed manual curation. Steps included removing exact duplicates, overly short texts, and examples consisting solely of URLs. Language filtering ensured all texts were in Portuguese. Crucially, instances with high textual similarity but conflicting truth labels were manually reviewed and resolved. External label verification using the Google FactCheck API also helped correct inconsistencies. Finally, all explicit URLs were removed from the texts to prevent models from learning biases from specific domains.
The Enrichment Process in Action
The enrichment process was adaptive. If an initial web search using the pre-processed text yielded a strong match (meaning 80% or more of the query terms were present in the highlighted snippets of the top five results), the claim extraction step was skipped. Otherwise, the LLM extracted a concise claim, which was then used for a second web search. A separate search was also performed using the Google FactCheck Claims Search API.
The need for claim extraction varied significantly across datasets: 94% for COVID19.BR, 80.1% for Fake.Br, and 70.7% for MuMiN-PT. This suggests that informal, fragmented messages like those from WhatsApp (COVID19.BR) often require summarization for effective searching, while datasets derived from already fact-checked news (MuMiN-PT) might be more directly searchable.
The LLM was effective in extracting claims, averaging 11-12 words and forming single, focused sentences. Analysis of the most frequent terms in these extracted claims showed they generally maintained the original topics but often focused on more specific entities or events, indicating effective summarization.
Insights from Search Results
The search results from the Google Custom Search Engine (CSE) predominantly pointed to reliable sources like government domains and major media outlets. Domains like Facebook appeared more frequently for datasets originating from social media, like COVID19.BR and MuMiN-PT. The presence of specific health-related domains for pandemic-era datasets also highlighted the search’s sensitivity to temporal and thematic contexts.
A portion of CSE results also linked to pages indexed by Google FactCheck, containing journalistic verifications. This was particularly relevant for MuMiN-PT (21.3% of returns), reinforcing its connection to pre-verified claims. The Google FactCheck Claims Search API showed varying success rates, with MuMiN-PT having the highest (58.8%) and Fake.Br the lowest (0.7%). This API also provided the truth labels assigned by fact-checking organizations, which predominantly classified claims as “false” or “misleading,” aligning with the primary focus of these agencies.
Qualitative Patterns in Enriched Data
For claims originally labeled as true, search results typically corroborated the information by linking to original news sources or other reliable outlets (Corroboration Pattern). Explicit confirmations by fact-checking agencies for true claims were rare, further supporting the observation that these agencies primarily focus on debunking false information.
For false claims, three patterns emerged: direct refutation by reliable sources, reinforcement of the fake news by unreliable sources (a problematic outcome for automated systems), and academic recognition of the fake news. The latter, where academic papers cited the fake news as a study example, served as an interesting indirect form of verification, validating the claim’s falsity within the research community.
Also Read:
- Automating Insights from Oral Histories with Language Models
- Adaptive Prompting Strategies Boost Biomedical Named Entity Recognition
Impact on Fake News Detection Models
The study experimentally evaluated the impact of data validation and enrichment on the performance of fake news detection models, specifically Bertimbau (a Portuguese-adapted BERT model) and Gemini 1.5 Flash. The validation process, while improving data quality, initially made the classification task more challenging by removing certain discriminative signals, leading to a slight performance decrease compared to original data.
However, the subsequent enrichment with external content generally improved performance over the validated-only data, especially for Bertimbau and Gemini on the COVID19.BR dataset. This suggests that adding external context provides valuable information. Interestingly, filtering out social media sources from the enriched data sometimes reduced performance, indicating that even less curated information can hold relevant signals.
Consistently, fine-tuning Bertimbau outperformed few-shot learning with Gemini 1.5 Flash. This aligns with existing literature, which suggests that with sufficient training examples, specialized smaller models can surpass larger, more general-purpose models in specific tasks.
This research demonstrates the feasibility of systematically enriching Portuguese fact-checking corpora with contextual evidence, providing more robust resources for combating misinformation. While challenges remain, such as reliance on commercial APIs and the dynamic nature of information, this methodology lays a solid foundation for future advancements in semi-automated fact-checking. You can read the full research paper here.
