TLDR: Hi-OSCAR is a new hierarchical open-set classifier for Human Activity Recognition (HAR) that accurately identifies known activities and intelligently rejects unknown ones. By organizing activities into a hierarchy, it can localize unknown activities to their closest known category, providing more insight than a simple ‘known/unknown’ label. The research also introduces NFI FARED, a diverse and challenging new dataset for HAR research, enabling more robust testing of open-set classification models. Hi-OSCAR demonstrates superior performance and a balanced approach to both known and unknown activity detection.
Human Activity Recognition (HAR) is a field focused on classifying activities using data from sensors, often worn on the body. While HAR has numerous applications in sports, medicine, and forensics, it faces a significant challenge: the vast difference between the activities people perform in real life and the limited number of activities captured in training datasets. Traditional HAR systems, known as closed-set classifiers, struggle when encountering activities they haven’t been trained on, often misclassifying them with high confidence. This limitation seriously undermines their reliability in real-world scenarios.
Addressing this, researchers have introduced Hi-OSCAR, a Hierarchical Open-set Classifier for Human Activity Recognition. This innovative model not only identifies known activities with high accuracy but also effectively rejects unknown activities. Crucially, it goes beyond a simple ‘known/unknown’ distinction by localizing unknown activities to the nearest internal node within a structured hierarchy of activity classes. This provides valuable insight into what an unknown activity might be similar to, rather than just labeling it as entirely novel.
The Hierarchical Approach
The core idea behind Hi-OSCAR is to organize activity classes into a structured hierarchy, recognizing that not all activities are equally dissimilar. For instance, ‘walking’ and ‘walking upstairs’ are more closely related than ‘walking’ and ‘punching’. This hierarchical arrangement embeds additional information into the model’s predictions. Hi-OSCAR traverses this tree-like structure through multiple classification decisions. When an unknown activity is encountered and rejected, the path taken through the hierarchy can still be inspected, revealing which known classes are most similar to the unseen activity. For example, an unknown ‘sitting’ activity might be localized near a ‘standing’ node, offering a more nuanced understanding.
Unlike existing hierarchies in other domains, HAR lacks a definitive, universally agreed-upon hierarchy. Hi-OSCAR tackles this by deriving its hierarchy using self-supervised clustering based on the statistical features of the activities themselves. This ensures the hierarchy reflects physical similarities in the data rather than just semantic ones, which might not always align with sensor readings (e.g., ‘standing’ and ‘elevator’ can be semantically distinct but physically similar).
Introducing NFI FARED: A New Dataset for HAR
To facilitate open-set HAR research, the team behind Hi-OSCAR collected a new, publicly available dataset called NFI FARED. This dataset is a significant contribution, featuring data from multiple subjects performing nineteen diverse activities across various contexts, including daily living, commuting, and rapid movements. With nearly 40 hours of data, NFI FARED offers a richer and more challenging environment for developing and testing HAR models compared to existing datasets, which often have fewer activities and less diversity. The dataset deliberately includes within-class variations, such as different speeds for stair climbing or sitting with/without a backrest, to broaden class definitions and reflect realistic intra-class diversity.
How Hi-OSCAR Works Under the Hood
Hi-OSCAR uses a CNN-based feature extractor, adapted from existing ResNet architectures, to process raw sensor signals from multiple body-worn devices. This extractor learns to identify patterns in the data. The generated hierarchy, created using Hierarchical Agglomerative Clustering (HAC), guides the classification process. A specialized loss function ensures that the model accurately classifies known activities while also maximizing uncertainty for activities that fall outside the known categories. During inference, a ‘mean path entropy’ score is calculated to determine the confidence of a prediction. If this confidence is too low, an ‘inference stopping criterion’ is triggered, and the model outputs an internal node in the hierarchy, providing an approximation of the unknown activity’s nature.
Also Read:
- Decoding Robot Mistakes: How Human Reactions Can Guide AI
- New Deep Learning Study Sets a Standard for Cricket Shot Classification, Reveals Reproducibility Challenges
Performance and Impact
Experiments show that Hi-OSCAR consistently outperforms state-of-the-art baselines in both identifying known activities and detecting unknown ones. On the challenging NFI FARED dataset, Hi-OSCAR demonstrated a significant performance gap over other models, highlighting its robustness. Crucially, it achieves a strong balance between in-distribution (known activity) accuracy and out-of-distribution (unknown activity) detection, a critical requirement for practical HAR applications. The ability to approximate unknown classes to internal nodes in the hierarchy provides a continuous measure of similarity, offering more meaningful feedback than a simple binary ‘known/unknown’ label.
The introduction of NFI FARED, with its diverse and challenging activities, opens new avenues for research in HAR, particularly in the area of near-OOD detection, where unknown activities are very similar to known ones. The Hi-OSCAR model, detailed further in the research paper, represents a significant step towards more reliable and informative human activity recognition systems that can adapt to the unpredictable nature of real-world human behavior.
