TLDR: This research paper investigates the limitations of Knowledge Graph-based Retrieval-Augmented Generation (KG-RAG) models, particularly their ability to reason with incomplete knowledge. It introduces a new benchmark and evaluation method that forces models to infer answers from indirect evidence. The findings reveal that current KG-RAG methods have limited reasoning capabilities when direct facts are missing, often relying on memorized information from textual labels rather than true symbolic reasoning. The study highlights the need for more robust retrieval and reasoning strategies in KG-RAG systems.
Knowledge Graph-based Retrieval-Augmented Generation (KG-RAG) is an exciting area in artificial intelligence, aiming to combine the powerful reasoning abilities of large language models (LLMs) with the structured, factual evidence found in knowledge graphs. This approach is designed to help LLMs answer questions and perform tasks using more comprehensive and up-to-date information than what they might have memorized during their initial training.
However, a recent research paper titled “What Breaks Knowledge Graph based RAG? Empirical Insights into Reasoning under Incomplete Knowledge” highlights significant shortcomings in how these KG-RAG systems are currently evaluated. The authors, Dongzhuoran Zhou, Yuqicheng Zhu, Xiaxia Wang, Hongkuan Zhou, Yuan He, Jiaoyan Chen, Evgeny Kharlamov, and Steffen Staab, point out two main issues. Firstly, many existing benchmarks include questions that can be answered directly by simply retrieving existing facts from the knowledge graph, making it unclear if the models are truly ‘reasoning’ or just performing a direct lookup. For example, if a knowledge graph contains the fact that ‘Justin Bieber has a brother named Jaxon,’ a question like ‘Who is Justin Bieber’s brother?’ doesn’t require complex inference. Secondly, inconsistent evaluation metrics and overly lenient answer matching criteria across different studies often inflate performance estimates, making it difficult to compare different KG-RAG methods meaningfully.
To address these challenges, the researchers introduce a novel method for constructing benchmarks and an evaluation protocol specifically designed to assess KG-RAG methods under conditions of incomplete knowledge. Their core idea is to create natural language questions whose answers are not explicitly stated in the knowledge graph but can only be found by logically inferring them through alternative paths. This ensures that models must genuinely reason rather than just retrieve direct evidence.
The benchmark construction involves a two-step process. First, high-confidence logical rules are mined from the knowledge graph to identify facts that are inferable. Then, a subset of these inferable facts is intentionally removed from the knowledge graph, while ensuring that enough supporting information remains for the answer to still be logically deduced. Natural language questions are then generated based on these removed facts, forcing the models to rely on reasoning. The study utilized two well-established knowledge graphs: the synthetic Family dataset and the real-world FB15k-237 dataset, allowing for evaluation across different complexities and domains.
The empirical study conducted using this new benchmark revealed several critical limitations of current KG-RAG systems. A significant finding is that most models struggle to find answers when direct supporting facts are removed, indicating their limited reasoning capacity. While methods that involve training (like RoG and GNN-RAG) showed more resilience to incomplete knowledge compared to non-trained systems, even they exhibited a substantial decline in performance when direct evidence was absent. This suggests that while training can help, current models still heavily rely on explicit information.
Another crucial insight from the research is the profound influence of entity labeling. When entities were represented by natural language labels (e.g., ‘Barack Obama’), models performed significantly better. This suggests that LLMs often leverage their internal, memorized knowledge associated with these text labels rather than performing symbolic reasoning over abstract identifiers. Surprisingly, using official entity IDs (like ‘/m/02mjmr’) provided almost no benefit over randomly assigned private IDs, indicating that LLMs treat these identifiers as opaque tokens unless a clear text label is provided.
The paper also includes case studies illustrating common failure patterns, such as models failing to retrieve relevant reasoning paths or generating incorrect answers even when the correct context was retrieved. These failures highlight the need for more advanced retrieval strategies that can identify indirect paths and improved reasoning modules that can better distinguish relevant from irrelevant information.
Also Read:
- OmniBench-RAG: A New Standard for Evaluating Retrieval-Augmented Generation
- Retrieval-Augmented Generation: A Comprehensive Review of Its Landscape
In conclusion, this work provides a valuable framework for evaluating KG-RAG systems under realistic conditions of knowledge incompleteness. The findings underscore that while KG-RAG is a promising direction, current methods have significant limitations in their true reasoning capabilities, often relying on memorization and struggling when direct evidence is unavailable. Future research should focus on developing more robust retrieval mechanisms, enhancing generalization in reasoning modules, and carefully crafting fine-tuning strategies to improve performance without compromising the LLM’s inherent reasoning ability. You can read the full paper here: What Breaks Knowledge Graph based RAG? Empirical Insights into Reasoning under Incomplete Knowledge.
