TLDR: EHWGesture is a new, large-scale multimodal video dataset designed to improve the understanding of clinical hand gestures. It features over 1,100 recordings from 25 healthy subjects performing five clinically relevant gestures, captured by RGB-Depth and event cameras, with precise ground-truth tracking from a motion capture system. The dataset also includes varying execution speeds to enable Action Quality Assessment, making it a valuable benchmark for advancing automated clinical assessments of hand dexterity.
Understanding hand gestures is a crucial area in human-computer interaction, with significant implications for various applications, including the automatic assessment of hand dexterity in clinical settings. While advancements in deep learning have greatly improved the recognition of static gestures, comprehending dynamic gestures remains a complex challenge. This complexity arises from the intricate spatiotemporal variations involved in movement, coupled with the limitations of existing datasets that often lack diverse multimodal and multi-view data, precise ground-truth tracking, and a component for evaluating the quality of an action.
Addressing these critical gaps, a new research paper introduces EHWGesture, a groundbreaking multimodal video dataset specifically designed for understanding clinical gestures. This dataset is a significant step forward, featuring five gestures that are highly relevant in clinical evaluations of hand dexterity. It encompasses more than 1,100 recordings, totaling approximately six hours of video, collected from 25 healthy individuals.
The data collection for EHWGesture is remarkably comprehensive. It utilizes two high-resolution RGB-Depth cameras, providing both color and depth information, alongside an advanced event camera. To ensure unparalleled accuracy, a motion capture system was employed to provide precise ground-truth tracking of hand landmarks. All recording devices were meticulously spatially calibrated and synchronized, guaranteeing seamless cross-modal alignment of the data. Furthermore, to integrate an action quality assessment (AQA) task directly into gesture understanding, the recordings are organized into different execution speed classes, mirroring the way hand dexterity is evaluated in clinical practice.
Key Contributions of EHWGesture
The EHWGesture dataset makes several vital contributions to the field:
- It provides a large-scale multimodal dataset, integrating synchronized and spatially calibrated RGB, depth, and event-based data.
- It offers high-quality ground truth through a motion capture system, enabling precise hand landmark tracking for accurate validation and benchmarking.
- It introduces controlled-speed recordings, guided by a metronome, which allows for Action Quality Assessment based on execution speed, similar to clinical evaluations.
- It establishes baseline models for gesture classification, temporal segmentation (trigger detection), and Action Quality Assessment.
- It investigates how different modalities, input sequence lengths, and frame rates impact gesture understanding.
The gestures included in the dataset are commonly used in clinical assessments based on the Unified Parkinson’s Disease Rating Scale (UPDRS). These include dynamic movements like finger tapping, hand opening and closing, hand pronation-supination, and finger-to-nose reaching, as well as a static gesture involving extending the arm forward to assess tremors. The inclusion of varying speeds for finger tapping, pronation-supination, and opening-closing gestures (SLOW, NORMAL, FAST) is particularly innovative, as slowness of movement (bradykinesia) is a key indicator in Parkinson’s disease, making execution speed a crucial factor for AQA.
Dataset Creation and Characteristics
The dataset was created with 25 healthy volunteers, ensuring ethical considerations were met with written informed consent and anonymized data. The instrumentation included an Optitrack motion capture system, two Microsoft Azure Kinect RGB-Depth cameras, and an Inivation DVXplorer Lite event camera. Temporal alignment was achieved using an OptiTrack eSync2, and extensive calibration procedures ensured spatial alignment across all devices.
Each subject performed the five tasks with both hands, with each task continuously recorded for 20 seconds, reflecting real-world clinical assessments. The timed gestures were performed at 75 bpm (SLOW), 115 bpm (NORMAL), and 145 bpm (FAST) using a metronome. Motion capture data provided reference points for temporal segmentation, allowing for efficient and accurate labeling of gesture triggers.
In total, the dataset comprises 1,100 recordings, resulting in 2,200 RGB-Depth videos and 1,100 event recordings. This amounts to over 2.6 million RGB and depth frames, and an additional 660,000 event frames when aggregated, bringing the total to over 3.3 million frames. The dataset supports combined gesture recognition and AQA tasks, grouped into 11 classes (two free gestures plus three gestures at three velocities).
Experimental Insights
Baseline experiments conducted using EHWGesture highlight its potential for gesture classification, gesture trigger detection, and action quality assessment. The researchers trained models for gesture classification (identifying the five gestures) and AQA (classifying gestures based on execution speed). They also focused on detecting trigger events within recordings, using motion capture data as ground truth.
The results showed that increasing the number of input modalities generally improved performance across both tasks. Fusing RGB and depth data from a single camera provided a modest boost, while incorporating data from two distinct cameras led to more substantial improvements. Leveraging all three modalities (RGB, depth, and event-based data) yielded the highest performance gains, with an average accuracy increase of 3.3% for gesture detection and 4.5% for quality estimation.
For trigger detection, the baseline approach achieved high detection accuracy, but precisely identifying the exact timing of the trigger event remains a challenge. Interestingly, slower movements benefited significantly from longer smoothing windows, which helped reduce false detections, suggesting that AQA predictions could be used to fine-tune trigger detection.
Also Read:
- Understanding Student Engagement: How Biosensors and Multimodal Analytics are Shaping Online Education
- Advanced Dental Imaging: A Multimodal Approach to Precise Tooth Segmentation
Future Directions
EHWGesture is poised to serve as a comprehensive benchmark for advancing multimodal clinical gesture understanding. It offers a rich resource for developing more sophisticated models for Action Quality Assessment, particularly for use with real pathological data. The dataset’s multi-view and multimodal nature opens up new avenues for research into how different perspectives and data types influence gesture understanding.
While the dataset currently features healthy subjects and has some limitations regarding background diversity and skin tone representation, plans are in place to extend it to include pathological subjects and improve diversity. The availability of this dataset, detailed in the paper available at EHWGesture – A dataset for multimodal understanding of clinical gestures, marks a significant step towards automated and more accurate clinical assessments of hand dexterity.
