TLDR: A research paper introduces an LLM-powered agent that allows users to interact with Enterprise Resource Planning (ERP) systems using natural language. It features a dual-agent architecture (reasoning and critique) to translate natural language queries into accurate SQL statements, improving reliability and efficiency in accessing complex ERP data. The system was tested on a production ERP database, showing promising results with open-weight LLMs, though it highlights challenges like the need for manual database schema descriptions and handling vague user prompts.
Enterprise Resource Planning (ERP) systems are the backbone of many businesses, integrating and managing crucial processes across departments like manufacturing, finance, and human resources. While essential for data-driven decisions, their complexity often requires users to have specialized knowledge of database structures to retrieve information.
Recent advancements in Artificial Intelligence, particularly Large Language Models (LLMs), are opening new doors for how we interact with these complex systems. Imagine being able to simply ask your ERP system a question in plain English, and it provides the answer. This is the core idea behind a new intelligent software agent designed to interface with industrial production-grade ERP systems.
The AI Agent: Bridging Language and Data
This innovative agent leverages open-weight LLMs to translate natural language queries into executable SQL statements, making ERP data more accessible. To ensure accuracy and reliability, the system employs a novel dual-agent architecture, combining a ‘reasoning’ stage with a ‘critique’ stage. This allows for iterative refinement and validation of the generated queries, significantly improving their precision based on partial result analysis.
The journey of converting natural language into SQL, known as Text-to-SQL, has seen rapid development thanks to LLMs. Early methods relied on rigid rule-based systems or traditional deep learning models. However, LLMs have shifted the paradigm, acting as zero- or few-shot agents that can perform semantic parsing through simple prompts, reducing the need for task-specific models.
While initial LLM-based Text-to-SQL approaches showed promise, they struggled with complex queries or ambiguous database structures. To overcome these hurdles, newer agentic architectures have emerged. These designs integrate LLMs with external tools and structured reasoning workflows, enhancing SQL generation without modifying the core model parameters. Such agentic designs offer several advantages: they reduce reliance on large annotated datasets, improve results through modular reasoning, and increase adaptability across unfamiliar databases.
How the Architecture Works
The system’s architecture is built around a REACT-Based (Reason + Act) conversational agent. When a user inputs a query, the REACT agent first interprets the intent. If it can be addressed by a database query, the task is delegated to a specialized SQL Agent.
The SQL Agent is where the magic happens, transforming natural language into optimized SQL. It operates with a two-agent collaborative loop: the Reasoner and the Critic. The Reasoner generates, executes, and refines SQL queries based on the user’s request and feedback. It even has a self-debugging strategy, analyzing error messages from failed executions to iteratively correct the query.
Complementing the Reasoner is the Critic agent. This internal evaluator assesses the quality, correctness, and efficiency of the SQL queries. It checks for syntactic validity, semantic appropriateness (ensuring the query truly answers the user’s intent), and query efficiency (identifying redundancies). If deficiencies are found, the Critic provides structured feedback to the Reasoner, initiating a refinement cycle.
To ensure the SQL Agent understands the database context, a structured schema is injected. This schema has two parts: a handcrafted semantic description by an expert developer, which explains tables, concepts, and relationships in natural language, and an auto-generated list of all tables, columns, data types, and key constraints.
A ‘Human-in-the-Loop’ (HITL) mechanism is also integrated. Before executing any database queries, the REACT agent can clarify user intent, preventing incorrect or vague prompts that might waste computational resources. If the intent is clear and answerable, the SQL query generation proceeds; otherwise, the user is prompted for clarification.
Also Read:
- Bridging the Gap: How AI Helps Access Manufacturing Knowledge Graphs
- Empowering Supply Chain Decisions with Large Language Models
Testing and Future Directions
The agent was evaluated using a production-grade SQL Server 2017 ERP database, containing 7 tables and 321 columns. The experiments involved 11 different open-weight LLMs, with Qwen2.5-32B handling multi-turn dialogue and LLaMA 3.1-8B extracting structured outputs. The agent was tested with 12 distinct queries, curated and validated by an experienced ERP system user.
Results showed promising performance, with models like Devstral 24B Q4 achieving high accuracy (10 out of 11 correct answers), followed by Qwen 2.5 32B Q4 (9 out of 11). However, some models struggled or failed due to hardware limitations and generation time constraints.
Implementing such a SQL agent in an ERP system involves several steps: exposing the database in read-only mode, documenting relevant fields, having an ERP expert create a natural language description of the system, and carefully selecting and configuring the LLMs (a two-model approach is recommended for performance).
While demonstrating the feasibility of integrating LLMs into industrial environments for natural language interaction with ERPs, the research also highlighted challenges. The agent currently requires manual alignment with the database through expert-written natural language descriptions, limiting its ‘plug-and-play’ scalability. Additionally, the agent sometimes struggles with vague prompts, making it difficult to select the correct columns, and the REACT agent occasionally fails to fully summarize user intent to the SQL Agent.
Future work aims to address these limitations, including developing a new agent to automatically describe the database, reducing the need for manual expert input, and improving methods for intent interpretation and column selection under uncertainty. For more details, you can read the full research paper here.
