TLDR: A new method called SECA (Semantically Equivalent and Coherent Attacks) generates realistic, meaning-preserving prompt variations that significantly increase Large Language Model (LLM) hallucination rates. Unlike previous unrealistic attacks, SECA uses LLMs to propose and verify semantically equivalent and coherent prompts, revealing that verbose and lexically diverse rephrasings can trick LLMs into producing incorrect and fabricated responses while appearing natural to humans.
Large Language Models (LLMs) are becoming increasingly common in critical areas like medical diagnosis and financial analysis. However, a significant concern is their tendency to “hallucinate” – producing incorrect or fabricated information. This unreliability poses serious risks, especially when these models are trusted uncritically.
Previous attempts to provoke these hallucinations, often called adversarial attacks, frequently resulted in unrealistic prompts. These prompts might include gibberish tokens or drastically alter the original meaning, offering limited insight into how hallucinations might occur in real-world scenarios. Unlike computer vision, where realistic modifications to images are common for adversarial attacks, finding realistic adversarial prompts for LLMs has been largely unexplored.
To bridge this gap, researchers from the University of Pennsylvania have introduced a novel method called Semantically Equivalent and Coherent Attacks (SECA). SECA aims to elicit LLM hallucinations through realistic modifications to prompts that preserve the original meaning and maintain linguistic coherence. This approach provides a more practical understanding of LLM vulnerabilities.
How SECA Works
SECA formulates the challenge of finding realistic attacks as a constrained optimization problem. Imagine trying to find a slightly different way to ask a question that still means the same thing, but somehow tricks the LLM into giving a wrong answer. That’s essentially what SECA does.
The method involves three key components:
- Attack Objective: The goal is to maximize the likelihood of the LLM generating a specific incorrect answer and a hallucinated explanation. For example, if the correct answer to “What is p in 24 = 2p?” is 12, SECA might try to make the LLM output 8 and then justify it with a fabricated explanation.
- Semantic Equivalence Constraint: This is crucial for realism. SECA ensures that the modified prompt means the same thing as the original prompt. A “feasibility checker LLM” acts as a judge, evaluating if the new prompt maintains mutual entailment, introduces no new information, omits no essential information, preserves the original meaning, and leads to the same ground-truth answer.
- Semantic Coherence Constraint: The modified prompt must also be linguistically natural and human-like. This is typically measured by perplexity, where lower values indicate greater coherence.
SECA uses a “proposer LLM” to generate multiple candidate prompts that are semantically equivalent to the original. These candidates are then filtered by the “feasibility checker LLM” to ensure they meet the semantic equivalence and coherence criteria. Finally, among the valid candidates, the one most likely to elicit the target hallucination is selected. This iterative process allows SECA to efficiently search for effective yet realistic adversarial prompts.
Key Findings and Implications
Experiments conducted on open-ended multiple-choice question answering tasks, using a filtered subset of the MMLU dataset, demonstrated SECA’s effectiveness across various open-source and commercial LLMs, including Llama, Qwen, and GPT models.
- Higher Attack Success Rates: SECA achieved significantly higher hallucination rates compared to raw prompts and other existing methods like Greedy Coordinate Gradient (GCG), which often produces incoherent prompts.
- Preserved Realism: Crucially, SECA maintained minimal violations of semantic equivalence and coherence constraints, meaning the adversarial prompts it generated were realistic and plausible to humans.
- Prompt Characteristics: The research found that more verbose and lexically diverse prompts were more likely to induce hallucinations. This suggests that subtle linguistic variations, even when preserving meaning, can obscure the core intent for LLMs and trigger failures.
- LLMs as Evaluators: The study also validated that LLM-based evaluators could reliably check for semantic equivalence and classify hallucination types, showing strong agreement with human annotators.
This research highlights a significant vulnerability: even slight, meaning-preserving rephrasing can cause LLMs to hallucinate. This underscores the importance of evaluating LLM robustness under realistic attack scenarios, especially as these models are deployed in high-stakes applications. The code for SECA is open-sourced, which you can find more about in the original research paper.
Also Read:
- Untargeted Jailbreak Attack: A New Approach to Uncover LLM Vulnerabilities
- Dynamic Target Attack: A New Strategy for Bypassing LLM Safety Alignments
Future Directions
The authors suggest several avenues for future work, including integrating gradient estimation techniques to accelerate SECA’s convergence for large-scale testing, extending the method to free-form generation tasks beyond multiple-choice questions, developing untargeted attack variants, and applying SECA to reasoning models that generate step-by-step explanations.
Ultimately, SECA serves as a vital tool for understanding and mitigating hallucinations, paving the way for more robust and trustworthy LLMs in real-world applications.
