TLDR: REMONI is an autonomous remote health monitoring system that combines wearable devices, IoT, and multimodal large language models (MLLMs). It continuously collects vital signs, accelerometer data, and video, detecting anomalies like falls and critical health changes. Medical professionals can interact with an AI assistant via a web app to get real-time patient status, activity, and emotion insights, aiming to reduce healthcare workload and costs.
In an era where wearable technology is increasingly common, the demand for effective remote patient monitoring has grown significantly. While much research has focused on collecting and analyzing sensor data for anomaly detection in specific diseases, there has been a notable gap in human-machine interaction within this field. Addressing this, a new autonomous system called REMONI has been proposed, aiming to bridge this gap by integrating multimodal large language models (MLLMs), the Internet of Things (IoT), and wearable devices.
What is REMONI?
REMONI, which stands for REmote health MONItoring, is designed to automatically and continuously collect vital signs and accelerometer data from wearables like smartwatches, as well as visual data from patient video clips captured by cameras. This collected data is then processed by an anomaly detection module, which includes a sophisticated fall detection model and algorithms to identify and alert caregivers to emergency conditions.
A key feature of REMONI is its natural language processing (NLP) component, powered by MLLMs. This allows the system to detect and recognize a patient’s activity and emotion, and to respond to inquiries from healthcare workers. Through a user-friendly web application, doctors and nurses can interact with an intelligent agent to access real-time vital signs, current patient state, and mood. The system aims to reduce the workload of medical professionals and potentially lower healthcare costs.
How Does It Work?
The REMONI system is built upon three main modules: Anomaly Detection, a Natural Language Processing Engine, and an Internet of Things module for deployment.
Anomaly Detection
This module is crucial for identifying irregular patterns that might indicate falls or critical changes in a patient’s health. For fall detection, the system uses a hybrid deep learning (HDL) model that combines convolutional neural networks (CNNs) for spatial feature extraction and long short-term memory (LSTM) networks for temporal sequence analysis. This model processes accelerometer data, primarily from the wrist, and has shown high accuracy in distinguishing falls from daily activities.
In addition to fall detection, REMONI includes a threshold-based algorithm to monitor five key vital signs: body temperature, heart rate, respiration rate, blood pressure, and oxygen saturation. If any of these values fall outside predefined healthy ranges, the system promptly sends an alert to medical personnel.
Natural Language Processing Engine
Medical professionals interact with REMONI through a web application. The NLP engine, which powers this interaction, operates in three stages: intention detection, data preparation, and final output production. Initially, a General Large Language Model (LLM) processes caregiver inquiries to understand their intent. Based on this, the engine retrieves necessary data from cloud storage or edge devices. It can also utilize an MLLM to describe the patient’s current activity and emotion from images, and a plotting function to visualize vital sign data. Finally, all gathered information is compiled, and the General LLM uses it to accurately answer the user’s questions.
Internet of Things System
The IoT system comprises sensors (smartwatches and cameras), edge devices, cloud storage, and cloud computing. Wearable applications on smartwatches collect accelerometer and physiological data, transmitting it to an edge device. The edge device processes this real-time data for anomaly detection and immediately sends alerts to the cloud computing platform if an emergency is detected. Otherwise, it periodically uploads vital signs and visual data to cloud storage for archiving. Cloud computing hosts the web application, receives emergency alerts, and facilitates communication between the NLP engine, edge devices, and cloud storage.
Experimental Results
Experiments demonstrated the system’s feasibility. The fall detection model, using accelerometer data from a Samsung Galaxy Watch 3, achieved a high accuracy of 98%. For activity and emotion recognition, various MLLMs were evaluated, with GPT4-Vision showing the best performance (51% accuracy for activity and 41% for emotion), despite some limitations that are being addressed with future fine-tuning.
The overall system response time for user queries was generally under 20 seconds, with emergency alerts from the edge device to caregivers averaging a rapid 0.2 seconds. This indicates a responsive and efficient system for critical situations.
Also Read:
- Social Robots That Endure: The Quest for Resilient AI
- MedAlign: A New AI Framework for Accurate and Efficient Medical Imaging Analysis
Conclusion
REMONI represents a significant step forward in remote health monitoring, offering a comprehensive solution for continuous data collection, anomaly detection, and seamless communication between medical professionals and an intelligent system. The system is designed to be scalable and adaptable, allowing for the integration of more wearable devices and anomaly detection algorithms. Its potential to alleviate the workload of medical professionals and reduce healthcare costs is substantial. For more details, you can refer to the full research paper here.
