TLDR: A new research paper introduces an automated method to create multi-turn Natural Language to Graph Query Language (NL2GQL) datasets using Large Language Models (LLMs). This addresses the challenge of real-world, context-dependent interactions with graph databases, which are typically multi-turn. The paper proposes a framework to build the MTGQL dataset, the first Chinese multi-turn NL2GQL dataset, and establishes baseline methods for evaluating multi-turn NL2GQL translation, highlighting the complexities of maintaining context across turns.
In the evolving landscape of data management, graph databases have become indispensable for representing and analyzing complex relationships across various domains. Interacting with these powerful systems typically requires specialized technical knowledge of Graph Query Languages (GQL). This often creates a barrier for many users who prefer to communicate in natural language.
Recent advancements have focused on translating natural language (NL) into GQL (NL2GQL). However, most existing methods are designed for single-turn queries, meaning they handle one question at a time without remembering past interactions. In real-world scenarios, users often engage in multi-turn conversations, iteratively refining their questions or seeking more details based on previous responses. This dynamic, context-dependent interaction is a significant challenge that single-turn approaches cannot effectively address.
Furthermore, the progress in multi-turn NL2GQL has been hindered by a severe lack of high-quality datasets specifically designed for these complex interactions. To tackle this critical issue, researchers have proposed an innovative automated method for constructing multi-turn NL2GQL datasets, leveraging the capabilities of Large Language Models (LLMs).
Introducing MTGQL: A New Multi-Turn Dataset
The core of this research is the development of the MTGQL dataset, which is built using the proposed automated method. This dataset is derived from a real-world financial market graph database and is set to be publicly released, providing a crucial resource for future research in this field. The paper also introduces three types of baseline methods to evaluate the effectiveness of multi-turn NL2GQL translation, laying a solid foundation for further advancements.
The Dependency-Aware Multi-Turn Dataset Construction Framework
The framework for building the MTGQL dataset is designed with a keen awareness of dependencies within a dialogue. It operates through five interconnected components:
-
Context Manager: This central component maintains the dialogue history, including previous questions, answers, entities, and relationships. It ensures that the system remembers past interactions and uses this context to guide subsequent queries. It also selects appropriate ‘expansion patterns’ (like asking for attributes, temporal shifts, or related entities) to ensure diverse and logical follow-up questions.
-
Question Generator: An LLM acts as the Question Generator, creating both initial and follow-up questions. These questions are designed to be colloquial, informal, and even ambiguous, mimicking real user queries. It uses placeholders for specific entities (e.g., “What is the opening price of [s] stock today?”) which are later filled by the Context Manager with actual data from the database.
-
GQL Generator: This component takes the natural language question and the database schema to generate the corresponding GQL query. It uses a fine-tuned LLM to ensure accuracy and adherence to the graph database’s structure.
-
GQL Validator and Optimizer: This crucial module ensures the generated GQL queries are both syntactically correct (executable) and semantically accurate (reflecting the user’s intent). It runs the query against the database and, if errors are found, uses the LLM to optimize and correct the GQL, repeating the process if necessary.
-
Dataset Filter: To prevent redundancy and ensure high quality, this component applies two filtering methods. It removes duplicate GQL structures by replacing entity names with placeholders and discards semantically similar questions using embedding-based comparisons.
The MTGQL dataset, constructed using this framework, contains 4,500 multi-turn dialogues, with an average of 6.49 turns per dialogue. It encompasses nearly 30,000 GQL statements, involving a rich diversity of entities and relations, making it a complex and challenging benchmark for models.
Also Read:
- AbsCon: Enhancing Graph Model Generation from Text with Large Language Models
- Navigating Knowledge Graphs: A New Framework for Efficient Question Answering
Evaluation and Future Directions
Human evaluations by domain experts confirmed the dataset’s high quality in terms of coherence, question diversity, coverage, and semantic accuracy. Unlike other existing NL2GQL datasets, MTGQL is uniquely designed for multi-turn interactions, making it a pivotal resource for advancing research in this area.
The research also benchmarked several methods, with the Dependency-aware (DA) method consistently outperforming others. However, the results indicate that maintaining context and generating accurate queries become significantly more challenging as the number of turns increases. This highlights the ongoing need for more sophisticated models capable of handling complex, chained questions and resolving ambiguities across multiple dialogue turns.
While this work marks a significant step forward, the researchers acknowledge limitations, including the current focus on Chinese language data and the reliance on manually designed schemas or dependency-aware modules. Future work aims to expand the dataset to be bilingual (Chinese-English), explore schema-agnostic approaches, and further automate the dataset construction process to reduce manual tuning.
This groundbreaking research provides a robust framework and a valuable dataset that will accelerate the development of more intelligent and user-friendly systems for interacting with graph databases through natural language. For more details, you can refer to the full research paper: Multi-turn Natural Language to Graph Query Language Translation.
