TLDR: MIRAGE-Bench is a novel benchmark designed to systematically identify and evaluate hallucinations in LLM-based agents operating in interactive environments. It introduces a three-part taxonomy for agentic hallucinations (unfaithful to task instructions, execution history, or environment observations) and employs a ‘contextual snapshot’ strategy for reliable elicitation. The evaluation uses an ‘LLM-as-a-Judge’ paradigm to assess agent actions. Findings indicate that hallucinations are a persistent challenge across models, with proprietary LLMs showing only a modest performance lead, and highlight common patterns of fabricating unsupported information.
Large language models (LLMs) are increasingly used as the core intelligence for advanced AI agent systems. These agents can perceive environments, make decisions, and interact with external systems in various domains like web automation, software engineering, and robotics. However, a critical limitation inherited from their underlying neural architecture is ‘hallucination’ – the generation of outputs inconsistent with their contextual input. Unlike traditional language generation tasks, hallucinations in agent settings directly translate into undesirable actions, significantly amplifying real-world risks due to the model’s autonomy and potential for human oversight.
While previous studies have highlighted these failures, existing evaluations have been fragmented and lacked a unified testing framework. To address this, researchers have introduced MIRAGE-Bench (Measuring Illusions in Risky AGEnt settings), the first unified benchmark designed to elicit and evaluate hallucinations specifically in interactive LLM-agent scenarios.
Categorizing Agent Hallucinations
MIRAGE-Bench introduces a three-part taxonomy to categorize agentic hallucinations, extending from prior work that distinguishes between content unfaithful to context versus factual knowledge. For LLM agents, hallucinations are defined as actions unfaithful to one or more of these components:
-
Task Instruction Unfaithfulness: Actions that violate the specified task objective, exceed constraints, or rely on unstated user intentions. For example, an agent might execute operations with a fabricated API set or act on extrapolated user intentions beyond the stated task.
-
Interaction History Unfaithfulness: Actions that conflict with the agent’s knowledge based on its past trajectory, such as repeating completed steps or ignoring prior outcomes.
-
Environment Observation Unfaithfulness: Actions that misrepresent or disregard the agent’s observed information of the environment states. This could involve clicking a non-existent button, misreading a file directory, or assuming a non-existent transition state.
It’s important to distinguish hallucinated actions from merely incorrect actions. While all hallucinated actions are incorrect, not all incorrect actions are hallucinations. Hallucinations specifically stem from unfaithfulness to contextual input, whereas other errors might arise from a lack of domain knowledge, flawed logic, or limited contextual memory.
Eliciting and Evaluating Hallucinations
To elicit these failures, MIRAGE-Bench performs a systematic audit of existing agent benchmarks, identifying common risk settings that trigger hallucinations. These settings are then used to synthesize new test cases. A key challenge in evaluating agent hallucinations is their stochastic nature and difficulty in reproduction. To overcome this, the benchmark adopts a ‘contextual snapshot’ strategy. This involves freezing the agent’s state immediately before a potential hallucination point, allowing the LLM to generate its next action within a controlled, deterministic context. This ‘freeze-frame’ approach stabilizes the evaluation environment and enables precise, scalable detection of deviations from expected behavior.
For evaluation, MIRAGE-Bench employs a fine-grained ‘LLM-as-a-Judge’ paradigm. This approach leverages the semantic reasoning capabilities of advanced LLMs (like o4-mini) to assess agent behavior. Unlike traditional natural language generation (NLG) tasks, agent actions are dynamic and context-dependent, making direct verification difficult. The LLM judge assesses whether the agent’s beliefs and decisions remain faithful to the context it received. Actions are categorized and scored as: ‘Faithful Action’ (score 1), ‘Incomplete Action’ (score 0.5), or ‘Hallucinated Action’ (score 0). Incomplete actions occur when the agent is neither clearly faithful nor overtly hallucinatory, such as continuing to explore without making progress on an infeasible task.
Also Read:
- Navigating the AI Frontier: Large Language Models in Social Simulation
- Assessing Voice Understanding in AI: Introducing the Speech Intelligence Quotient
Key Findings and Insights
The evaluation of twelve prominent LLMs (both open-source and proprietary) using MIRAGE-Bench revealed several critical insights:
-
Persistent Challenge: Despite architectural differences and scale, all evaluated models exhibited relatively low utility scores and substantial hallucination rates. Even top proprietary models frequently produced unfaithful actions, underscoring the ongoing severity of hallucination issues.
-
Modest Lead from Proprietary Models: While proprietary models generally outperformed open-source counterparts, the performance gap was modest. This suggests that scaling and advanced instruction tuning alone do not fully resolve faithfulness issues, highlighting the need for targeted alignment methods specific to interactive agent contexts.
-
Presumptive Hallucination Pattern: A recurring issue observed was LLM agents fabricating contextually unsupported information, similar to chatbot-style generation. For instance, if a required button was not observed, the agent might assume its existence and attempt to click it. This behavior likely stems from instruction tuning in open-domain dialogue settings, which encourages speculative completions—an inductive bias misaligned with the demands of agentic environments.
-
Pop-up Distractions: Interestingly, stronger models showed slightly higher rates of interaction with pop-up distractions, while weaker models almost never did. This suggests that increased perceptual capacity might introduce mild susceptibility to irrelevant cues. However, most models showed surprising robustness to text-based pop-ups, indicating that optimizing the modality and structure of agent observations might alleviate certain risks.
In conclusion, MIRAGE-Bench provides a systematic approach to studying hallucinations in interactive LLM agents. The findings underscore that hallucination remains a persistent and widespread challenge, emphasizing the need for targeted alignment techniques beyond simple scaling or instruction tuning. The research paper and dataset are available for further exploration at https://arxiv.org/pdf/2507.21017.
