TLDR: EHR-ChatQA is a novel interactive benchmark designed to evaluate LLM-powered agents for Electronic Health Record (EHR) data access. It addresses real-world challenges like query ambiguity and value mismatch through simulated user interactions across two flows: Incremental Query Refinement (IncreQA) and Adaptive Query Refinement (AdaptQA). Experiments show a significant gap between optimistic (Pass@5) and consistent (Pass^5) agent performance, highlighting a critical lack of robustness in current models for safety-critical EHR applications. The benchmark provides tools for schema and value exploration, web search, and SQL execution, offering insights into common failure modes and guiding future agent development.
Large Language Models (LLMs) are increasingly being used as autonomous agents to tackle complex tasks, especially when interacting with structured databases. This capability is particularly valuable in critical fields like Electronic Health Records (EHRs), where enabling natural language queries over vast patient data can significantly improve clinical research and patient care. However, despite their impressive performance, the adoption of LLM-powered agents for EHR data access has been limited by the absence of benchmarks that truly reflect real-world clinical data access scenarios.
Researchers have identified two main challenges in deploying these agents: query ambiguity, where user questions are often vague (e.g., “Show me recent labs” without specifying types or timeframes), and value mismatch, where clinical terms used by users may not align with database entries (e.g., “Lopressor” vs. “metoprolol tartrate”). Existing text-to-SQL benchmarks, which focus on translating a single, well-formed question into an SQL query, do not adequately capture these complexities.
To address this critical gap, a new interactive database question answering benchmark called EHR-ChatQA has been introduced. This benchmark evaluates the entire workflow of database agents, from clarifying user questions and using tools to resolve value mismatches, to generating correct SQL and delivering accurate answers. EHR-ChatQA places an agent in a simulated environment with an LLM-based user and a suite of tools, providing a comprehensive evaluation of agent capabilities in interactive clinical data access flows.
The benchmark covers diverse patterns of query ambiguity and value mismatch through two distinct interaction flows:
Incremental Query Refinement (IncreQA)
In IncreQA, users progressively add new constraints to their existing queries. This tests the agent’s ability to maintain conversational context and integrate new details without losing prior information. Examples include adding a patient ID to a patient cohort or specifying event timings.
Also Read:
- KGQAGen: A Framework for High-Quality Knowledge Graph Question Answering Datasets
- VitaBench: A New Standard for Evaluating LLM Agents in Real-World Scenarios
Adaptive Query Refinement (AdaptQA)
AdaptQA assesses an agent’s capacity to reformulate its query plan when users modify their search goals mid-conversation. This could involve searching for alternative medications if the initially requested one is not found, or adopting fallback strategies when no relevant data is available. These tasks often require more advanced value mismatch resolution.
The EHR-ChatQA benchmark is grounded in real-world clinical QA scenarios and utilizes two publicly available EHR databases (MIMIC-IV and eICU). To ensure that agents rely on schema exploration tools rather than memorized schemas, all table and column names in these databases have been renamed.
The benchmark provides agents with a set of default tools for schema exploration (e.g., `table_search`, `column_search`), value exploration (e.g., `value_substring_search`, `value_similarity_search`), external knowledge retrieval (`web_search`), and SQL execution (`sql_execute`). A simulated LLM-based user, designed with a certain level of stochasticity, initiates vague queries, forcing the agent into dialogue for clarification. A dedicated validator ensures the reliability of these simulations by flagging and rerunning invalid interaction traces.
Experiments with state-of-the-art LLMs, including o4-mini and Gemini-2.5-Flash, across five independent trials revealed a significant finding: while agents achieved high optimistic success rates (Pass@5 of 90–95% on IncreQA and 60–80% on AdaptQA, meaning success in at least one of five trials), their consistent success rate (Pass^5, meaning success across all five trials) was substantially lower by 35–60%. This critical robustness gap highlights a lack of reliability in current agents, which is a major concern for deployment in safety-critical domains like EHRs.
Further analysis of the top-performing models, such as o4-mini, showed that this inconsistency widens with more trials and is negatively correlated with the number of user interactions. Common failure modes include Value Linking Errors (failing to retrieve all relevant database terms, like drug brand names or synonyms), SQL Generation Errors (subtle flaws in query logic, such as misinterpreting event timing), Rule Violations, and Limited Clinical Knowledge. Inconsistent failures were primarily driven by SQL errors, where slight variations in user phrasing could disrupt the agent’s context tracking, leading to incorrect SQL generation.
The authors, Gyubok Lee, Woosog Chay, Heeyoung Kwak, Yeong Hwa Kim, Haanju Yoo, Oksoon Jeong, Meong Hi Son, and Edward Choi, emphasize that future agent development should focus on improving exploration strategies to address value mismatch and enhancing context management techniques to ensure consistent state-tracking across linguistic variations. This research provides valuable diagnostic insights to guide the creation of more performant and reliable agents for interactive EHR QA.
The code for EHR-ChatQA is available at https://github.com/glee4810/EHR-ChatQA, and the full research paper can be accessed here: https://arxiv.org/pdf/2509.23415.
