TLDR: JARVIS is a new AI framework that uses two Large Language Models (LLMs) to make HVAC systems easier to interact with for non-experts. It translates user questions into data queries, processes real-time sensor data, and generates clear answers. By intelligently handling domain-specific knowledge and complex data operations, JARVIS significantly improves the accuracy and helpfulness of responses compared to existing methods.
A new research paper introduces JARVIS, an innovative framework designed to enhance how we interact with smart Heating, Ventilation, and Air Conditioning (HVAC) systems. This system aims to make complex HVAC data accessible and understandable for everyone, not just experts, by using advanced Large Language Models (LLMs).
The smart building HVAC industry is experiencing significant growth, with projections reaching $8.31 billion by 2029. However, interacting with these systems often requires specialized knowledge to query databases and interpret sensor information. Traditional methods, like human experts, are costly and not scalable, while rule-based systems struggle with the nuances of natural language.
This is where LLMs come in. They offer the scalability of automation combined with the flexibility of natural language understanding. However, integrating real-time sensor data, which is constantly updated, poses a unique challenge for LLMs trained on static information. The paper highlights that raw sensor data often needs domain-specific processing, like statistical summaries, before an LLM can use it effectively.
Introducing JARVIS: A Two-Stage LLM Framework
JARVIS addresses these challenges with a two-stage, master-worker architecture. It consists of an Expert-LLM and an Agent. The Expert-LLM acts as the “master,” interpreting high-level user questions and translating them into precise instructions. The Agent, acting as the “worker,” then executes these instructions, performing tasks like retrieving data from databases, processing it, and finally generating a natural language response.
Key innovations in JARVIS include:
- Adaptive Context Injection: JARVIS intelligently categorizes and injects contextual information. HVAC-common knowledge (like reasoning patterns) is fine-tuned into the Expert-LLM, while dynamic, deployment-specific details (like sensor layouts or user preferences) are injected in real-time via prompting. This ensures the system has both deep domain understanding and up-to-date information.
- Parameterized SQL Builder and Executor: Instead of relying on LLMs to generate complex SQL queries directly (which can be error-prone), JARVIS’s Expert-LLM provides high-level query intentions. A rule-based module then constructs the complete, syntactically correct SQL statements, including necessary filtering and joins. This makes data access more reliable and robust.
- Bottom-up Planning: To ensure coherent multi-stage responses, the Expert-LLM first defines an “expected answer template.” It then plans the preceding data querying and processing steps backward from this desired final output, ensuring all necessary information is gathered and presented consistently.
Overcoming Text-to-SQL Limitations
A preliminary study conducted by the researchers revealed common pitfalls in generic text-to-SQL models when applied to HVAC data. These included difficulties with user-native language, fragility in generating complex SQL, and issues with handling missing sensor values. JARVIS tackles these by decoupling high-level intent from low-level SQL generation and by using a Python-based data processing pipeline for complex mathematical and logical operations that LLMs typically struggle with.
How JARVIS Works in Practice
When a user asks a question, the Expert-LLM first uses its “Thinking Component” to interpret the query, resolve ambiguities, and outline a plan. It then generates structured JSON instructions for the Agent. The Agent’s “Query Execution Module” retrieves data, and the “Data Processing Module” (using Python’s Pandas library) performs necessary computations. Finally, a general-purpose LLM, guided by the processed data and the Expert-LLM’s “User Expectation Component,” generates the final, user-friendly response.
Also Read:
- Automating Data and AI Workflows with the Data Agent Architecture
- Unlocking Advanced AI Reasoning: A New Framework for Smarter Language Models
Evaluation and Performance
JARVIS was evaluated using real-world sensor data from a commercial HVAC system and a dataset curated by HVAC experts. The evaluation used both an “LLM-as-a-Judge” framework (where other LLMs assessed response quality) and a user study with 22 participants. JARVIS consistently outperformed baseline systems across metrics like cohesiveness, helpfulness, and truthfulness.
The study showed that each component of JARVIS significantly contributes to its overall performance. For instance, the adaptive context injection and thinking module help interpret ambiguous user queries, while the query execution and data processing modules ensure accurate and reliable data handling. Interestingly, JARVIS also demonstrated efficient performance, being one of the fastest configurations despite its sophisticated architecture, thanks to optimized planning by the Expert-LLM.
While JARVIS represents a significant step forward, the researchers acknowledge areas for future improvement, such as enhancing the robustness of structured output generation and exploring different LLM architectures. This work paves the way for more intuitive and effective interaction with complex sensor-driven systems. For more detailed information, you can refer to the full research paper here.
