TLDR: This paper introduces an agent-based system that allows industrial machines to be controlled using natural language commands, moving beyond traditional touch-based interfaces. It leverages large language models (LLMs) equipped with specialized tools and the OPC UA communication standard to interact with Programmable Logic Controllers (PLCs). The system was evaluated on a Siemens S7-1500 PLC, demonstrating high accuracy (90.0% to 98.0%) across various LLMs in executing commands to read, write, or adjust machine parameters. The approach aims to simplify human-machine interaction in manufacturing environments, making it more intuitive and efficient.
In the rapidly evolving landscape of modern manufacturing, automated machines are becoming increasingly prevalent. While these machines excel at repetitive tasks, human operators often step in for knowledge-based interactions and decision-making that go beyond predefined rules. This highlights the critical role of the human-machine interface (HMI), especially in complex industrial settings where fast and natural interaction is key.
Traditional HMIs often rely on touch-based graphical user interfaces (GUIs), which can involve deep menu navigation and require extensive operator training. This can be time-consuming and less efficient in daily operations. However, with significant advancements in deep learning and large language models (LLMs), HMIs are now transitioning towards natural user interfaces (NUIs). NUIs allow operators to interact with computer systems using natural language, similar to how we interact with consumer devices like smartphones.
The Challenge in Industrial Settings
While natural language interaction is common in consumer technology, industrial applications present unique challenges. Machines are expensive, potentially dangerous, and often operate in noisy environments. These factors necessitate robust theoretical frameworks and empirical investigations to ensure the safe and effective deployment of such technologies in factories.
A Novel Agent-Based Approach
A recent research paper, titled “Beyond touch-based HMI: Control your machines in natural language by utilizing large language models and OPC UA”, proposes an innovative agent-based approach to address these challenges. This method aims to modify machine parameters through natural language, potentially supplementing or even replacing current touch-based HMIs in industrial settings. The core of this approach involves equipping LLMs with specialized tools and integrating them with the Open Platform Communications Unified Architecture (OPC UA) communication standard.
The system allows operators to issue commands in natural language, such as “Please decrease the temperature by 20 % in machine 1 and set the motor speed to 5000 rpm in machine 2.” The LLM receives this input, interprets it, and then selects one of three predefined tools: a reading tool, a writing tool, or an adjustment tool. These tools connect to an OPC UA server, which in turn communicates with a Programmable Logic Controller (PLC) to either change or read the value of a specific machine parameter (known as a node).
The LLM is guided by a system prompt that defines its role and provides essential context, including a dictionary of accessible machine variables (with their OPC NodeIds and data types) and machine credentials for establishing a secure connection. The tools themselves are Python functions that encapsulate the logic for connecting to the OPC UA server and performing operations. By offloading calculations to these tools, the risk of LLM hallucinations is reduced, and smaller, less computationally intensive models can be utilized.
Experimental Validation and Results
To evaluate this conceptual approach, the researchers set up an experiment using a Siemens SIMATIC S7-1500 PLC with four exemplary parameters: motorspeed, temperature, textfield1, and textfield2. A local workstation hosted an MCP server and ran the OPC UA and LLM client. The study utilized three proprietary LLMs (GPT-5, GPT-5 mini, GPT-5 nano) and two open-weight models (GPT-oss:20b, Qwen3:32B).
Fifty synthetically generated commands, varying in complexity from single to multiple parameter changes, were used to test the models. The accuracy was measured based on whether all affected parameters were correctly set. The results were highly promising: proprietary GPT models achieved accuracies between 96.0% and 98.0%, while open-weight models reached up to 90.0%. Interestingly, the high-performance GPT-5 model did not significantly outperform its smaller proprietary counterparts, suggesting that smaller models can be effective for this task.
The study also categorized five types of errors, with the most common being incorrect tool execution due to sign errors (e.g., increasing instead of decreasing a value). Other errors included repeated tool execution, uncertain interpretations requiring user confirmation, misinterpretation of operation verbs, and misinterpretation of the tool itself.
Also Read:
- Unpacking the Architecture of Autonomous LLM Agents
- ManiAgent: Orchestrating Robot Actions with AI Agents
Conclusion and Future Directions
This research demonstrates a viable agent-based approach for controlling industrial machines through natural language, offering a more intuitive and efficient alternative to traditional touch-based HMIs. The universal design of the approach means it can be applied to any machine supporting the OPC UA standard. The high accuracy achieved by both proprietary and open-weight LLMs underscores the potential of this technology.
Future research will explore more complex environments with a greater number of parameters and investigate triggering complete actions, such as starting/stopping machines or initiating calibration routines, rather than just interacting with single parameters. This work represents a significant step towards advancing natural interaction in industrial human-machine interfaces. You can read the full research paper here.
