TLDR: A new research paper explores the current state and challenges of integrating Large Language Models (LLMs) directly into relational database management systems (DBMS) for advanced querying. The study evaluates open-source and enterprise solutions using five representative LLM queries, identifying key issues in enforcing structured outputs, optimizing resource utilization, and improving query planning. It highlights that while enterprise solutions show promise, open-source systems struggle with functionality and performance, emphasizing the need for tighter LLM+DBMS integration for scalable and efficient processing of LLM queries.
Large Language Models (LLMs) have rapidly become indispensable tools for a variety of applications, from summarizing text to answering complex questions. More recently, these powerful AI models are being integrated directly into relational database management systems (DBMS) to revolutionize how we query and process data. Major players like Amazon, Databricks, Google, and Snowflake are already offering LLM invocation within SQL, a feature known as LLM queries, to unlock deeper data insights.
However, the journey to seamlessly blend LLMs with databases is not without its hurdles. Open-source solutions, in particular, currently face significant limitations in both functionality and performance. A recent research paper, Research Challenges in Relational Database Management Systems for LLM Queries, delves into these issues, providing an early exploration of existing systems and highlighting critical areas for improvement.
The Core Problem: Bridging the AI-Database Divide
The paper, authored by Kerem Akillioglu, Anurag Chakraborty, Sairaj Voruganti, and M. Tamer Özsu from the University of Waterloo, reveals that current LLM+DBMS integrations struggle with fundamental aspects. For instance, processing a mere 17,000 rows with a simple LLM query using local inference on a single A100 GPU and an open-source DBMS took a staggering 5 hours. Scaling this to a million rows would extend to approximately 12 days, underscoring a severe efficiency gap.
Beyond performance, privacy is a major concern. Enterprise LLM inference often exposes sensitive data to third-party providers. Organizations aiming to leverage LLM capabilities while maintaining data confidentiality must serve models locally, which then exacerbates the performance challenges.
Evaluating Current Systems with Representative Queries
To understand the landscape, the researchers evaluated two open-source systems—PostgreSQL with pgAI and DuckDB with FlockMTL—alongside an enterprise platform, MotherDuck. They used five representative LLM queries, each designed to test different aspects of LLM integration within SQL:
- Q1: LLM Projection – Invokes the LLM in the SELECT clause to transform and project textual data.
- Q2: LLM Filter – Uses the LLM in the WHERE clause to filter rows based on semantic criteria.
- Q3: Multi-LLM Invocation – Combines two LLM calls, one in SELECT and one in WHERE, for refined outputs.
- Q4: LLM Aggregation – Calls the LLM to assign satisfaction ratings, which are then aggregated (e.g., averaged).
- Q5: RAG (Retrieval-Augmented Generation) – Retrieves relevant context via similarity search before the LLM generates an answer.
Functional Roadblocks and Solutions
The study found that while Q1 (LLM Projection) generally succeeded, queries 2, 3, and 4 often failed in open-source systems due to issues with structured outputs. LLMs, by nature, can generate inconsistent text. If a query expects a strict “Yes” or a numerical score between 1 and 5, any deviation (like extra text) causes the query to fail. MotherDuck, leveraging OpenAI’s GPT-4o-mini, performed better here due to its constrained decoding capabilities.
The researchers addressed this by modifying FlockMTL to use vLLM’s structured decoding, which helped with Q2 and Q3. However, Q4 still failed due to a mismatch between the LLM’s output (an untyped text blob) and the database’s need for a structured numeric type for aggregation functions like AVG(). Query 5 (RAG) also presented query planning challenges, often leading to out-of-memory errors in DuckDB due to inefficient index lookups.
Performance Bottlenecks and Optimizations
Performance testing revealed significant latency differences. Open-source systems were orders of magnitude slower than MotherDuck. FlockMTL’s initial batching strategy, which tried to send many rows in a single prompt, proved inefficient. Modifying it to process one row at a time ensured completion but resulted in poor GPU utilization and slow runtimes.
PostgreSQL with pgAI faced issues with parallel execution and subtransactions. Forcing single-process execution resolved the error but was slow. Implementing batching optimizations in pgAI significantly improved performance, reducing per-row processing time and leading to more stable, higher GPU utilization.
MotherDuck’s superior performance is attributed to its use of the OpenAI API for inference and its ability to concurrently send up to 256 requests, effectively parallelizing processing and reducing latency.
Key Research Challenges for the Future
The paper identifies three main research challenges:
1. Structured Outputs
Ensuring LLMs generate syntactically consistent outputs is crucial. The paper discusses three approaches: fine-tuning (costly, schema-dependent), prompt engineering (probabilistic, context-specific), and constrained decoding (effective but adds computational overhead). The researchers found constrained decoding to be the most practical initial solution.
2. Resource Utilization
Efficient batching strategies are paramount. Frequent data transfers between CPU and GPU reduce utilization. The ideal solution involves asynchronous parallel requests to the inference engine, ensuring GPU activity overlaps with CPU operations. Additionally, separating batches for constrained and non-constrained generation can prevent performance degradation.
3. Query Plans
Current database optimizers treat LLM functions as “blackboxes,” ignoring the significant cost of LLM invocations. Future optimizers need to factor in LLM prompts, CPU-GPU resource allocation, execution ordering, batching, and KV-cache reuse. By understanding the true cost of LLM calls, optimizers can reorder input rows to maximize prompt prefix sharing and reduce pre-fill overhead, unlocking substantial performance gains.
Also Read:
- Boosting Database Performance: How Synthetic SQL Queries Are Training Smarter Cost Models
- Unlocking Insights from Large Tables with Natural Language Query Plans
Conclusion
Integrating LLMs into relational DBMS offers exciting possibilities for enhanced analytics. However, achieving efficient and practical execution, especially with local inference for privacy, presents considerable challenges. This early exploration highlights the critical need for tighter LLM+DBMS integration, focusing on structured outputs, optimized batching, and intelligent query planning, to truly harness the power of AI within our databases.
