TLDR: This research paper details a bi-encoder model for SemEval-2025 Task 7, focusing on multilingual and cross-lingual fact-checked claim retrieval. The model, fine-tuned from pre-trained transformers, uses lightweight architectures (under 500M parameters) and achieves 92% Success@10 in multilingual and 80% Success@10 in cross-lingual tasks, ranking 10th and 5th respectively. It leverages English translations and ensemble methods to handle diverse languages, including unseen ones, demonstrating effective performance on modest hardware.
In an era where information spreads globally at an unprecedented pace, the challenge of combating online disinformation across multiple languages has become critical. The SemEval-2025 Task 7, titled “Multilingual and Crosslingual Fact-Checked Claim Retrieval,” directly addresses this issue by aiming to automate the process of finding previously verified claims across different languages.
The core idea is to help fact-checkers efficiently identify and retrieve relevant fact-checks for social media posts, regardless of the language they are written in. This task is divided into two main categories: Multilingual Retrieval, where the social media post and the fact-check are in the same language, and Crosslingual Retrieval, where they are in different languages, requiring robust techniques to bridge language barriers.
A Lightweight and Efficient Approach
The research paper presents an approach that treats this problem as a Learning-to-Rank task, utilizing a bi-encoder model. This model is fine-tuned from pre-trained transformer models, specifically optimized for understanding sentence similarity. A key highlight of this method is its focus on lightweight models, all having fewer than 500 million parameters, making them efficient enough to be trained even on readily available hardware like Kaggle T4 GPUs.
The system works by processing social media posts and fact-checked claims through separate encoders, which convert them into dense vector representations. The similarity between these representations is then calculated using a temperature-scaled cosine similarity function. For training, the model employs a combination of MNR Loss and Vanilla Cross-Entropy Loss to ensure that correct matches are ranked higher.
Handling Language Diversity
The dataset for this task is extensive, covering fact-checked claims and social media posts in eight languages: French, Spanish, English, Portuguese, Thai, German, Malay, and Arabic. To truly test the model’s generalization capabilities, the evaluation phase introduced two entirely new, unseen languages: Polish and Turkish. This emphasizes the need for approaches that can adapt to new linguistic contexts without direct prior training.
To tackle the multilingual challenge, the model was trained using both the original source languages and their English translations. For cross-lingual retrieval, the strategy primarily relied on English translations to maintain consistency and improve performance across different language pairs. The researchers also employed techniques like layer freezing and gradient checkpointing to balance computational efficiency with retrieval effectiveness.
Key Models and Performance
The system leveraged several powerful pre-trained models as its backbone, including:
- The NovaSearch/stella-en-400M-v5 model, known for balancing scalability and accuracy.
- The intfloat/multilingual-e5-large-instruct model, a robust choice supporting 100 languages, built on xlm-roberta-large.
- The mixedbread-ai/mxbai-embed-large-v1 model, a state-of-the-art English embedding model.
These models were chosen for their performance and their ability to operate within the lightweight parameter constraint.
The results demonstrate the effectiveness of this approach. The system achieved a 92% Success@10 rate in multilingual retrieval, meaning that in 92% of cases, at least one correct fact-check appeared in the top 10 retrieved results. In the cross-lingual track, it achieved an 80% Success@10 rate. These scores placed the system 10th in the multilingual track and 5th in the cross-lingual track among all participating teams.
While the model showed strong performance in languages like Thai and French, it faced challenges with morphologically complex languages such as Polish and Turkish, where semantic similarity can be harder to capture. The study also highlighted risks like false positives due to surface similarities between claims, even when factual variations exist.
Also Read:
- Unpacking AI’s Factual Accuracy: A Deep Dive into Language Model Fact-Checking
- On-Premise Fact-Checking: A New Era of Accessible AI Verification
Future Directions
The research concludes that while multilingual embedding models are effective, their performance varies across languages, especially in low-resource environments. Future work aims to enhance the stability of cross-lingual retrievals by incorporating more data for less-resourced languages, fine-tuning for specialized domains, and combining various modeling strategies. This will contribute to building more robust real-world fact-check retrieval systems.
For more detailed information, you can read the full research paper here.
