TLDR: A new research paper introduces ADMIT, a few-shot knowledge poisoning attack that effectively manipulates RAG-based fact-checking systems. ADMIT injects minimal, semantically aligned malicious content into knowledge bases, tricking LLMs into producing attacker-controlled outputs with deceptive justifications. It achieves high success rates across various LLMs and retrievers, outperforming previous attacks, and is difficult to detect by current defenses, revealing significant fragilities in AI fact-checking.
In the rapidly evolving landscape of artificial intelligence, Retrieval-Augmented Generation (RAG) systems have emerged as powerful tools, enhancing Large Language Models (LLMs) by integrating external knowledge. This integration helps LLMs overcome limitations like outdated information, hallucinations, and gaps in domain-specific knowledge. RAG systems are widely used in various applications, from ChatGPT plugins to Bing Search, and are particularly crucial in fact-checking to combat misinformation.
However, this reliance on external knowledge sources introduces a significant vulnerability: knowledge poisoning. This is an attack where malicious content is injected into the knowledge base, tricking LLMs into generating attacker-controlled outputs that appear to be grounded in manipulated context. While previous research has highlighted LLMs’ susceptibility to misleading content, real-world fact-checking scenarios present a unique challenge because credible evidence typically dominates the information pool.
A new study introduces a novel approach to this problem called ADMIT (ADversarial Multi-Injection Technique). This method extends knowledge poisoning to the fact-checking setting, where retrieved context often includes authentic supporting or refuting evidence. ADMIT is a few-shot, semantically aligned poisoning attack designed to flip fact-checking decisions and induce deceptive justifications. Remarkably, it achieves this without requiring access to the target LLMs, retrievers, or even token-level control.
The core idea behind ADMIT is to generate and iteratively refine adversarial passages under a simulated verification setup. It uses ‘proxy verifiers’ and ‘proxy passages’ to mimic the target fact-checking environment. This allows the attacker to craft malicious content that is not only highly relevant to the query but also semantically aligned with existing credible information, making it incredibly difficult to distinguish from legitimate content. The attack also employs an ‘adversarial prefix augmentation’ technique to ensure that the injected malicious passages are ranked among the top retrieval results, even when strong counter-evidence is present.
Extensive experiments have demonstrated ADMIT’s effectiveness and transferability. It successfully transfers across 4 different retrievers, 11 LLMs, and 4 cross-domain benchmarks. The attack achieved an impressive average success rate (ASR) of 86% at an extremely low poisoning rate of 0.93 × 10-6. This means a tiny amount of injected malicious content can significantly alter fact-checking outcomes. Furthermore, ADMIT proved robust even when faced with strong counter-evidence, outperforming prior state-of-the-art attacks by an average of 11.2% across all settings.
One of the most concerning aspects of ADMIT is its ability to craft misinformation-level passages. Unlike older attacks that produce unreadable text or overtly malicious instructions, ADMIT generates semantically coherent, human-readable content that mimics journalistic tone and interweaves truth with falsehood. This makes it exceptionally challenging for both humans and automated systems to detect. The study found that nearly all ADMIT-generated passages were misclassified as ‘real’ by LLM-based fake news detectors, reflecting their high surface credibility.
The research also explored potential defenses against ADMIT, including statistical detection methods (like perplexity and ROUGE-N similarity), LLM-based knowledge consolidation techniques, and agent-based verification systems. Unfortunately, these defenses largely proved ineffective. Statistical methods failed to distinguish between clean and injected passages, and knowledge consolidation often amplified the adversarial influence. Even sophisticated ReAct agents, designed for structured reasoning, remained highly vulnerable, with attack success rates rising significantly as more adversarial passages were injected.
Also Read:
- Unmasking Hidden Threats: Dual-Trigger Backdoors in Graph Foundation Models
- The Silent Threat: Poisoning Risks in LLM Prompt Optimization
The findings of this study expose significant vulnerabilities in real-world RAG-based fact-checking systems. They highlight that factual robustness does not automatically follow the scale or reasoning ability of LLMs. The research underscores the urgent need for more advanced defenses that can track information provenance, assess uncertainty, and reason beyond mere surface consistency to protect against sophisticated knowledge poisoning attacks. For more in-depth technical details, you can refer to the full research paper.
