TLDR: DeKeyNLU is a new dataset of 1,500 annotated question-SQL pairs designed to improve how AI models break down tasks and extract keywords from natural language queries. When used to fine-tune the User Question Understanding module of the DeKeySQL framework, it significantly boosts SQL generation accuracy on benchmark datasets like BIRD and Spider. DeKeySQL, a RAG-based pipeline with user question understanding, entity retrieval, and generation/revision modules, achieves state-of-the-art accuracy and efficiency, making database access simpler for non-technical users.
In today’s data-driven world, accessing information from complex databases often requires specialized technical skills, particularly knowledge of SQL (Structured Query Language). This can be a significant barrier for non-technical users. Natural Language to SQL (NL2SQL) systems aim to bridge this gap by allowing users to ask questions in plain English and have them automatically converted into precise SQL commands. This innovation promises to make database interaction as simple as having a conversation.
Recent advancements in artificial intelligence, especially with techniques like Retrieval-Augmented Generation (RAG) and Chain-of-Thought (CoT) reasoning, have greatly improved NL2SQL performance. However, these systems still face significant hurdles. Two major challenges are accurately breaking down a user’s question into smaller, manageable tasks (task decomposition) and precisely identifying key terms (keyword extraction). Errors in these initial steps often lead to incorrect or inefficient SQL queries.
Introducing DeKeyNLU: A New Dataset for Smarter NL2SQL
To tackle these persistent issues, researchers have introduced DeKeyNLU, a novel dataset designed to enhance how AI models understand natural language for NL2SQL tasks. DeKeyNLU consists of 1,500 carefully annotated question-and-answer pairs. These pairs are specifically focused on refining task decomposition and improving keyword extraction accuracy, which are crucial for the RAG pipeline that powers many modern NL2SQL systems.
The creation of DeKeyNLU involved a meticulous process. It was derived from the BIRD dataset, a large and widely used benchmark in NL2SQL research. Initial annotations for task decomposition and keyword extraction were performed by an advanced AI model, GPT-4o. However, recognizing the limitations of even powerful AI, human experts then rigorously reviewed and corrected these initial outputs through a three-phase cross-validation process. This human-in-the-loop approach ensured the high quality and precision of the DeKeyNLU dataset, making it a reliable resource for training and evaluating NL2SQL models.
DeKeySQL: A Comprehensive Framework for Accurate SQL Generation
Building upon the insights from DeKeyNLU, the researchers also propose DeKeySQL, a new RAG-based NL2SQL pipeline. This framework is designed to overcome common problems like slow processing times, high operational costs, and accuracy limitations found in existing approaches. DeKeySQL operates through three distinct modules:
- User Question Understanding (UQU): This is the first step where the system comprehends the user’s natural language query. It uses models fine-tuned with the DeKeyNLU dataset to perform two critical functions: task decomposition, breaking down complex questions into main and sub-tasks, and keyword extraction, categorizing terms into ‘objects’ (table/column names) and ‘implementations’ (filtering criteria).
- Entity Retrieval: Once keywords are extracted, this module identifies relevant database elements such as table names, column names, specific values, and their descriptions. It employs advanced techniques like embedders, retrievers, and re-rankers to efficiently find the most pertinent information from the database.
- Generation: This final module is responsible for creating the SQL statement. It first generates an initial SQL query using a large language model, guided by the decomposed tasks and retrieved entities. Crucially, DeKeySQL includes a revision mechanism. If the initial SQL contains errors, the system feeds the erroneous query and error messages back to the AI for correction, ensuring the final output is syntactically correct and operational.
Also Read:
- Beyond Syntax: GBV-SQL Validates User Intent in Text-to-SQL Generation
- Enhancing LLM Accuracy with InfoGain-RAG: A New Approach to Document Filtering and Reranking
Impressive Results and Practical Implications
Experiments with DeKeySQL demonstrated significant improvements in SQL generation accuracy. Fine-tuning the User Question Understanding module with DeKeyNLU boosted accuracy on the BIRD development dataset from 62.31% to 69.10%. Similarly, on the Spider development dataset, accuracy improved from 84.2% to 88.7%. These results highlight the profound impact of the DeKeyNLU dataset in refining the core understanding capabilities of NL2SQL systems.
The research also revealed interesting insights into model selection: larger models like GPT-4o-mini excelled at complex task decomposition, while smaller models such as Mistral-7B proved more effective for precise keyword extraction. This suggests that tailoring model choices to specific sub-tasks can optimize overall performance. An in-depth analysis showed that the User Question Understanding module was the most significant factor influencing overall SQL generation accuracy, followed by entity retrieval and the revision mechanisms.
Beyond accuracy, DeKeySQL also showcased superior practical utility. It achieved state-of-the-art performance among evaluated open-source NL2SQL methods, demonstrating a strong balance of efficiency and accuracy. For instance, compared to other advanced methods, DeKeySQL achieved a substantial reduction in runtime and operational costs, making it highly promising for industrial applications. You can read the full research paper for more details at arXiv:2509.14507.
In conclusion, DeKeyNLU and DeKeySQL represent a significant step forward in making database interactions more intuitive and accessible for everyone. By focusing on meticulous data annotation and a well-structured AI pipeline, this work paves the way for more accurate, efficient, and user-friendly natural language interfaces for databases.
