TLDR: A new benchmark, AmbiGraph-Eval, evaluates how Large Language Models (LLMs) handle ambiguous queries in graph databases. The study categorizes ambiguity into attribute, relationship, and attribute-relationship types, and introduces a new metric (AREA) to assess performance. Findings show that even top LLMs struggle with these ambiguities, highlighting a need for better ambiguity detection and query syntax generation capabilities in future models.
Large Language Models (LLMs) have shown remarkable abilities in converting everyday language into database queries, particularly for complex graph-structured data. However, real-world questions often contain hidden uncertainties, and the interconnected nature of graph databases can make these challenges even more difficult, potentially leading to incorrect or unintended results.
To address this critical issue, a new research paper titled “AmbiGraph-Eval: Can LLMs Effectively Handle Ambiguous Graph Queries?” introduces a systematic way to evaluate how well LLMs handle these tricky situations. Authored by Yuchen Tian, Kaixin Li, Hao Chen, Ziyang Luo, Hongzhan Lin, Sebastian Schelter, Lun Du, and Jing Ma, the paper highlights a significant gap in current LLMs’ ability to manage ambiguity.
The researchers propose a clear classification of graph-query ambiguities, breaking them down into three main types: Attribute Ambiguity, Relationship Ambiguity, and Attribute-Relationship Ambiguity. Each of these is further divided into ‘Same-Entity’ and ‘Cross-Entity’ scenarios. For instance, Attribute Ambiguity occurs when a query refers to an entity’s characteristic, but multiple possible characteristics exist in the graph, like asking for a paper’s “citation count” when there are counts from different sources like Google Scholar and Semantic Scholar. Relationship Ambiguity arises when the specific connection needed to answer a query is unclear, such as asking for an architect’s “contributions” which could mean design or project contributions. Attribute-Relationship Ambiguity combines both, like asking for a movie’s “rating” which could be an internal score or external reviews.
To test LLMs rigorously, the team developed AmbiGraph-Eval, a unique benchmark dataset. This benchmark consists of 560 carefully crafted ambiguous queries, each paired with expert-verified answers in the form of graph queries (specifically, Cypher). The dataset was built through a two-stage process involving data collection from real databases, synthesis using LLMs, and a thorough human review to ensure genuine ambiguity and valid answers.
The study evaluated nine prominent LLMs, including both closed-source models like GPT-4o, GPT-4, Claude-3.5-Sonnet, and Qwen-Plus, and open-source models such as Qwen-2.5, LLaMA-3.1, DeepSeek-V2.5, and Mistral-7B. To accurately measure performance, the researchers introduced a new metric called Ambiguity-Resolved Execution Accuracy (AREA). This metric goes beyond just checking syntax; it verifies if the generated Cypher code actually compiles and retrieves all correct results from the database, especially when multiple interpretations of an ambiguous query are possible.
The findings from the AmbiGraph-Eval benchmark are insightful. Even the most advanced LLMs struggle significantly with ambiguous graph queries. While GPT-4o generally showed the best overall performance, particularly in cross-entity attribute and relationship ambiguity tasks, it still exhibited some weaknesses, for example, in handling same-entity relationship ambiguity. LLaMA-3.1 also performed strongly in certain areas, excelling in relationship ambiguity and same-entity attribute-relationship ambiguity. The study revealed that strong general reasoning abilities in LLMs do not automatically translate to superior performance in this specific task. Instead, a deep understanding of graph ambiguity and mastery of graph database query syntax are more crucial.
The research identifies several key challenges faced by current models: accurately detecting ambiguous intent, generating syntactically correct Cypher Query Language (CQL), correctly interpreting task instructions and graph structures, performing numerical calculations, and prioritizing comparisons. Ambiguity detection and query syntax generation were highlighted as major hurdles.
Also Read:
- Uncovering the Gaps: Why Knowledge Graph RAG Models Struggle with Incomplete Information
- Unmasking LLM Vulnerabilities with ObfusQAte
This work underscores a critical area for future development in LLMs. The authors suggest that future research should focus on improving models’ capabilities in both resolving ambiguity and generating correct syntax simultaneously. Techniques like syntax-aware prompting and explicitly highlighting ambiguities could be promising avenues. For more technical details, you can read the full research paper available at arXiv.org.
