TLDR: BenchPress is a human-in-the-loop system that uses LLMs and retrieval-augmented generation to quickly and accurately create domain-specific text-to-SQL benchmarks from enterprise SQL logs. It significantly reduces the manual effort and time required for annotation, outperforming traditional and vanilla LLM methods, especially for complex and private enterprise data, thus enabling better evaluation of LLMs in real-world business settings.
The paper introduces BenchPress, a new system designed to make it much faster and easier to create high-quality benchmarks for text-to-SQL models, especially for large private enterprise databases. Text-to-SQL models are large language models (LLMs) that translate natural language questions into SQL queries. While these models have shown great success on publicly available datasets like Spider, Bird, and Fiben, they often struggle when applied to the complex, domain-specific, and private data found in real-world enterprise settings.
The core problem BenchPress addresses is the difficulty and cost of manually annotating SQL logs with corresponding natural language questions. This task is usually performed by highly trained database administrators, which is both challenging and expensive. BenchPress tackles this by using a “human-in-the-loop” approach. It leverages retrieval-augmented generation (RAG) and LLMs to suggest multiple natural language descriptions for a given SQL query. Human experts then review, select, rank, or edit these suggestions, ensuring accuracy and alignment with the specific domain.
BenchPress was evaluated using annotated enterprise SQL logs, and the results showed that this LLM-assisted annotation significantly reduces the time and effort needed to create high-quality benchmarks. The combination of human verification with LLM-generated suggestions improved annotation accuracy, benchmark reliability, and the robustness of model evaluations. This system helps researchers and organizations assess how well text-to-SQL models perform on their unique, domain-specific data workloads.
Why is this important for enterprises?
Enterprise databases are very different from the public datasets typically used for training LLMs. They often feature complex and ambiguous schemas, integrate data from various systems, and use highly specialized terminology. Furthermore, enterprise SQL logs contain sensitive and proprietary information, making them unsuitable for public sharing. This means companies risk deploying text-to-SQL models that might fail on their specific data due to a domain mismatch. BenchPress provides a solution by allowing organizations to create custom benchmarks securely and efficiently, without needing to share their private data.
How BenchPress Works
BenchPress operates primarily through SQL-to-NL generation. Given a SQL query, it generates natural language descriptions. The system’s workflow involves a one-time setup and a repeated annotation loop. During setup, users configure their project, ingest SQL logs and schema files (either private or from supported public benchmarks like Beaver), and select the language model to be used.
In the annotation loop, for complex nested SQL queries, BenchPress can optionally decompose them into simpler subqueries. It then retrieves semantically similar examples and relevant schema information to provide context to the LLM. The LLM then generates four candidate natural language descriptions for each query or subquery. Human annotators then provide feedback by ranking, refining, or discarding these suggestions. Finally, the annotations can be reviewed and exported in a benchmark-ready format.
Key features of BenchPress include Retrieval-Augmented Generation (RAG) to improve relevance by embedding similar SQL queries and their annotations into prompts, prompt engineering and refinement allowing annotators to guide re-generation, query decomposition for complex SQL, and continuous human-in-the-loop feedback to ensure high-quality, enterprise-standard annotations.
Also Read:
- Advancing Conversational Database Interaction with MTSQL-R1’s Agentic Approach
- Optimizing NL2SQL with LLMs: Introducing the Datalake Agent for Cost-Efficient Database Interaction
Study Results
A user study involving 18 participants compared BenchPress against manual annotation and using a vanilla LLM (like ChatGPT) without specialized support. BenchPress consistently outperformed both other methods in terms of annotation accuracy, speed, and semantic fidelity. For instance, on the complex Beaver enterprise dataset, BenchPress achieved 86.1% accuracy compared to 66.2% for the vanilla LLM and 60.1% for manual annotation. It also drastically reduced annotation time, taking an average of 16.1 minutes for the Beaver dataset, while manual annotation took 102.1 minutes. The study highlighted that BenchPress is particularly effective with complex enterprise datasets, where other methods struggled significantly.
In conclusion, BenchPress offers a practical and scalable solution for enterprises to create their own domain-specific text-to-SQL benchmarks. This is crucial for accurately evaluating and deploying LLM-based text-to-SQL systems in real-world business environments. You can learn more about this system by reading the full research paper available at arXiv.org.
