TLDR: DailyLLM is a novel system that leverages multi-modal sensors on smartphones and smartwatches, combined with lightweight Large Language Models (LLMs), to automatically generate detailed and context-aware activity logs. It uniquely integrates location, motion, environment, and physiological data. The system outperforms existing methods in accuracy and efficiency, achieving a 17% improvement in log generation precision and nearly 10x faster inference speed. Designed for local deployment, DailyLLM enhances user privacy and can efficiently summarize activity data even on resource-constrained devices like the Raspberry Pi.
In our increasingly connected world, smart devices like smartphones and smartwatches have become ubiquitous. These devices are constantly collecting a wealth of data about our daily lives. Researchers are now exploring how to leverage this data to create detailed, context-aware activity logs, which can be incredibly valuable for understanding user behavior, monitoring health, and even creating ‘digital memories’ for individuals, including those with cognitive impairments.
However, existing methods for generating these activity logs often fall short. Many rely on manual input, which is cumbersome for users, or employ high-resolution cameras, raising significant privacy concerns and device costs. Furthermore, current automated systems often lack accuracy, efficiency, and the ability to capture the rich semantic context of activities.
Introducing DailyLLM: A Comprehensive Approach
To address these challenges, researchers from the University of California San Diego have developed DailyLLM, a groundbreaking system that automatically generates and summarizes context-aware activity logs. What sets DailyLLM apart is its comprehensive integration of contextual information across four key dimensions: location, motion, environment, and physiology. Crucially, it achieves this using only sensors commonly found in everyday smartphones and smartwatches.
DailyLLM is designed to be lightweight and efficient, utilizing a sophisticated framework that combines structured prompting with advanced feature extraction. This allows Large Language Models (LLMs) to achieve a high level of activity understanding without requiring massive computational resources.
How DailyLLM Works
The system operates in three main stages:
First, it involves **Multi-modal Data Collection and Processing**. DailyLLM gathers raw data from various sensors, including GPS, accelerometers, gyroscopes, magnetometers, barometers, Wi-Fi, Bluetooth, microphones, light sensors, temperature sensors, galvanic skin response (GSR), and photoplethysmography (PPG). This diverse data is then formatted, synchronized, and semantically annotated. For example, raw light, audio, and temperature readings are transformed into meaningful semantic levels like “Extremely dark,” “Normal Sound,” or “Comfortable” temperature, making them more interpretable.
Next is **Context-Aware Activity Understanding**. DailyLLM extracts key features from the processed sensor data, converting complex time-series information into structured representations that LLMs can easily interpret. A dynamic prompt generator then embeds these features into context-aware prompts, guiding the LLM to reason about the activity. This includes inferring activity types (e.g., walking, sitting, lying), understanding the scene (e.g., cafe, library, office), and describing the precise location, leveraging the LLM’s vast general knowledge.
Finally, **Activity Log Generation and Summarization** takes place. DailyLLM constructs detailed activity logs by integrating the inferred activity contexts with the semantic annotations. Over predefined time windows (e.g., 2 hours), it generates high-level summaries of user activities, including movement trajectories, changes in activity types, environmental conditions, and physiological trends. The system can also detect abnormal patterns, such as prolonged inactivity or unusual physiological signals, and provide personalized, conversational reminders to the user, like suggesting movement after a long period of sitting.
Key Innovations and Performance
DailyLLM represents a significant leap forward in life logging. It is the first sensor-based system known to automatically generate activity logs and summaries across all four dimensions: location, motion, environment, and physiology. The system’s innovative feature extraction and prompt engineering strategies significantly enhance the LLMs’ ability to interpret multi-modal sensor data.
Extensive experiments demonstrate DailyLLM’s superior performance. It achieves a remarkable 17% improvement in activity log generation BERTScore precision compared to state-of-the-art (SOTA) baselines, even while using a significantly smaller 1.5-billion parameter LLM model (Deepseek-R1-1.5B) compared to a 70-billion parameter SOTA model. This efficiency translates to nearly a 10x faster inference speed.
Beyond log generation, DailyLLM also excels in specific tasks. It achieves 100% accuracy in recognizing 15 distinct environmental scenarios and boasts a 92.46% accuracy in generating location descriptions, a task where even advanced general language models like GPT-4o struggle. The quality of its summaries is also rated as “Good” based on G-Eval scores.
One of DailyLLM’s most impressive features is its ability to be deployed locally on personal devices, including resource-constrained platforms like the Raspberry Pi 5. This local deployment is crucial for preserving user data privacy. On a Raspberry Pi, DailyLLM can summarize two hours of activity data within four minutes, highlighting its practical potential for real-world applications.
Also Read:
- Generative AI Enhances Realism in Human Motion Simulation
- AI Models Offer New Insights into Bridge Health Through Non-Destructive Evaluation
Conclusion
DailyLLM offers an effective and efficient solution for automatically recording and summarizing human activities. By integrating multi-modal sensor data with the advanced reasoning capabilities of LLMs, it provides rich, context-aware insights into daily life, paving the way for more personalized health interventions, lifestyle analysis, and the creation of valuable digital memories, all while prioritizing user privacy and computational efficiency.
