TLDR: A new survey explores the critical challenge of generalizability in IMU-based Human Activity Recognition (HAR), reviewing 229 papers and 25 datasets. It categorizes methodologies into model-centric (supervised, weakly supervised, unsupervised, self-supervised, and LLM-based learning) and data-centric (multi-modal fusion, cross-modal learning, and data augmentation) approaches. The paper also highlights current applications and future directions, including the integration of foundation models, generative AI, federated learning, and energy-efficient inference, aiming to overcome limitations like domain shifts and data scarcity for broader real-world adoption.
Human Activity Recognition (HAR) using Inertial Measurement Units (IMUs) is a rapidly growing field, crucial for the advancement of Wearable AI. IMUs, commonly found in devices like smartphones and smartwatches, capture detailed human motion data, enabling applications from fitness tracking to health monitoring. While HAR performance has seen significant improvements in controlled settings, its ability to generalize across different users, sensor placements, and environments remains a major hurdle for real-world adoption.
A recent comprehensive survey, titled Towards Generalizable Human Activity Recognition: A Survey, delves into the evolving landscape of IMU-based generalizable HAR. Authored by Yize Cai, Baoshen Guo, Flora Salim, and Zhiqing Hong, this survey reviews 229 research papers and 25 publicly available datasets, offering a broad and insightful overview of the field.
Understanding the Challenge of Generalization
The core problem in IMU-based HAR is the ‘domain shift.’ This occurs when models trained on data from one group of users, devices, or environments perform poorly when applied to new, unseen conditions. For instance, recognition accuracy can drop significantly when faced with variations in how different people perform the same activity, or when sensors are placed on different body parts. The survey highlights six distinct training settings that researchers use to tackle this challenge: within-group, cross-person, cross-device, cross-position, cross-activity, and cross-dataset HAR. Each setting represents a progressively more difficult scenario for models to generalize effectively.
Methodologies for Generalizable HAR
The survey categorizes existing approaches into two main perspectives: model-centric and data-centric.
Model-Centric Approaches
These methods focus on enhancing the models themselves to improve generalizability and reduce reliance on large amounts of labeled data. They include:
- Supervised Learning: Traditional methods that use labeled data. Recent advancements here involve ‘feature disentanglement’ to separate relevant activity features from irrelevant variations (like device placement), ‘multi-task learning’ to train models on related tasks simultaneously, and ‘federated learning’ which allows models to learn from decentralized data across many devices without sharing raw data, thus preserving privacy.
- Weakly Supervised Learning: Addresses situations where supervision signals are limited. This includes ‘semi-supervised learning’ (using both labeled and unlabeled data), ‘active learning’ (where the model intelligently selects the most informative unlabeled data to be labeled), and ‘inexact supervised learning’ (dealing with coarse or ambiguous labels, often using Siamese Networks to compare data pairs).
- Unsupervised Learning: Aims to discover patterns in data without any labels. ‘Clustering analysis’ groups similar activities, while ‘association analysis’ finds hidden correlations and temporal patterns.
- Self-Supervised Learning (SSL): A powerful paradigm where models learn from raw sensor data by solving ‘pretext tasks’ (e.g., predicting transformations applied to a signal or reconstructing masked parts of a signal). This helps models learn robust, transferable representations that can be fine-tuned for specific HAR tasks with less labeled data. Techniques like ‘transformation recognition,’ ‘reconstruction,’ and ‘contrastive learning’ are prominent here.
- LLM-based Learning: A cutting-edge area exploring the use of Large Language Models (LLMs) to enhance HAR. By integrating IMU data with LLMs, models can leverage the LLMs’ semantic understanding and reasoning capabilities, enabling zero-shot or few-shot recognition of new activities and even generating synthetic IMU data.
Data-Centric Approaches
These strategies focus on improving the data itself to enhance model performance and generalizability:
- Multi-Modal Fusion: Combines IMU data with other sensor modalities like video, audio, or physiological signals. This provides richer contextual information, leading to more accurate and robust HAR models.
- Cross-Modal Learning: Leverages richly labeled datasets from other modalities (e.g., video) to train models for IMU data, especially when IMU data is scarce. This involves translating knowledge between different sensor types.
- Data Augmentation: Techniques to expand the training dataset and introduce variability, helping models handle unseen conditions and prevent overfitting. This includes generating synthetic sensor sequences or applying transformations to existing data.
Datasets, Tools, and Applications
The survey also summarizes 25 widely used datasets, categorized into IMU-only and multimodal datasets, which are crucial for benchmarking and evaluating HAR algorithms. It also highlights various tools and benchmarks that support data preprocessing, model development, and standardized evaluation in the field.
IMU-based HAR has a wide range of applications, including healthcare and rehabilitation (e.g., fall detection, progress monitoring), sports and fitness monitoring (e.g., tracking exercise intensity), occupational work assessment (e.g., task efficiency), smart homes and assisted living (e.g., context-aware services), transportation and mobility (e.g., mode detection), human-robot interaction, and augmented/virtual reality (AR/VR) interactions.
Also Read:
- Internal Sensors Enable Robot Tactile Gesture Recognition
- Evaluating AI Hallucinations in First-Person Video Analysis
Future Outlook
Despite significant progress, challenges remain, such as obtaining reliable annotations, achieving efficient personalization under domain shifts, recognizing complex and composite activities, addressing privacy and security concerns, and standardizing evaluation protocols. The survey outlines several promising future directions:
- Foundation Models and LLMs in HAR: Training large models on multimodal sensor data and grounding LLMs in physical sensor data to enable more accurate and context-aware activity understanding.
- Cross-Modal and Generative AI: Using generative models to synthesize realistic sensor data for augmentation and to adapt models across different sensor domains.
- Federated and Continual Learning: Developing distributed learning ecosystems that ensure privacy, personalization, and continuous model improvement on user devices.
- Energy Efficiency and Resource-constrained Inference: Research into adaptive sensing and model compression to enable HAR systems to operate efficiently on wearable devices with limited resources.
This comprehensive survey serves as a valuable resource for researchers and practitioners, guiding them through the complexities and opportunities in advancing generalizable IMU-based Human Activity Recognition for widespread real-world impact.
