TLDR: Graph-RFT is a novel AI framework that enhances large language models (LLMs) in answering complex questions by reasoning over both knowledge graphs (KGs) and the web. It employs a two-stage process: first, Chain-of-Thought fine-tuning activates structured planning, and then reinforcement learning guides adaptive retrieval scheduling. This allows the model to decompose complex questions, identify missing facts, and dynamically combine KG and web searches, leading to superior performance even with smaller LLMs, especially when KGs are incomplete.
In the rapidly evolving world of artificial intelligence, answering complex questions accurately remains a significant challenge, especially when relying on vast, structured knowledge graphs (KGs). While large language models (LLMs) have shown impressive reasoning abilities, they often struggle to fully utilize the rich information in KGs and adapt when that information is incomplete. This can lead to fragmented reasoning and incorrect answers, even when the necessary facts might exist elsewhere.
A new research paper introduces a novel framework called Graph-RFT, which stands for “Plan Then Retrieve: Reinforcement Learning-Guided Complex Reasoning over Knowledge Graphs.” This innovative approach aims to empower LLMs to perform autonomous planning and intelligently decide when and how to retrieve information from both knowledge graphs and the broader web, particularly when KGs are missing crucial details.
Addressing Key Challenges in Question Answering
Current LLM-based methods for Knowledge Graph Question Answering (KGQA) typically fall into two categories: semantic parsing, which translates questions into executable queries, and retrieval-augmented generation (RAG), which uses KG triples to guide LLM responses. Both paradigms face limitations. They often assume KGs are complete, which is rarely the case in real-world scenarios. When information is missing, these methods lack a reliable way to find external facts. Furthermore, they frequently struggle with multi-step reasoning, failing to maintain a coherent plan across several steps, leading to errors.
How Graph-RFT Works: A Two-Stage Approach
Graph-RFT tackles these issues with a unique two-stage reinforcement fine-tuning process:
1. Reasoning Activation through CoT Fine-Tuning: The first stage involves a Chain-of-Thought (CoT) fine-tuning method. This trains the LLM using a specially designed dataset that includes step-by-step planning, reasoning, and retrieval processes. This initial training helps the model develop strong planning and reasoning capabilities and prepares it for the more advanced reinforcement learning stage.
2. Reasoning Enhancement with Reinforcement Learning: In the second stage, Graph-RFT employs a reinforcement learning method guided by a plan-retrieval strategy. This stage integrates explicit planning and retrieval actions directly into the model’s reasoning loop. The model learns a “coverage-aware” retrieval policy, meaning it intelligently coordinates between searching the knowledge graph and searching the web when the KG is incomplete.
The framework uses a structured template that allows the model to alternate between symbolic planning and retrieval actions. Inspired by Cartesian principles, a planning module breaks down complex questions into smaller, logically ordered sub-questions. Logical expressions then guide the use of various tools:
- Relation Search Tool: Helps find potential relationships for an entity within the KG.
- Neighbor Search Tool: Retrieves specific facts (tail entities) based on an entity-relation pair from the KG.
- Web Search Tool: Automatically activated when the KG lacks sufficient information, allowing the model to gather supplementary evidence from the internet.
To optimize this entire process, Graph-RFT uses a multi-reward system. This system includes an “outcome reward” for factual correctness and a “retrieval-specific reward” that evaluates how well and efficiently the model retrieves information. This unique reward structure encourages the model to learn precisely when and how to combine KG and web retrieval effectively, overcoming the limitations of previous methods.
Also Read:
- Structured Answers: Interpretable QA Using Knowledge Graphs
- Boosting LLM Performance with Strategy Fusion: Introducing SMaRT
Impressive Results and Future Implications
Experiments on several KGQA benchmarks demonstrate that Graph-RFT consistently outperforms existing state-of-the-art methods. Remarkably, it achieves superior accuracy even when using smaller LLM backbones, such as the Qwen2.5-7B series, outperforming models based on GPT-4 in many scenarios. The framework significantly improves complex question decomposition, factual coverage, and the coordination of different search tools.
Graph-RFT’s ability to adapt to incomplete knowledge graphs by intelligently leveraging web search is a major breakthrough. It shows that by combining structured planning with adaptive retrieval, AI systems can achieve more robust and accurate reasoning, making them more reliable for real-world applications. This research is a significant step towards integrating LLMs, KGs, and web applications, addressing a key scientific challenge in the field of semantics and knowledge. You can read the full paper here: Plan Then Retrieve: Reinforcement Learning-Guided Complex Reasoning over Knowledge Graphs.
