TLDR: A new research paper introduces “HannesImitation,” an AI-driven approach that uses imitation learning to enable the Hannes prosthetic hand to grasp objects more intuitively in various real-world scenarios. By learning from demonstrations, the system controls wrist movement and hand closure, achieving high success rates (79.3% overall) and demonstrating strong generalization to unseen objects (76% success). This method significantly outperforms traditional visual servoing in challenging human-to-prosthesis handovers, showcasing the potential of AI for enhancing prosthetic dexterity and reducing user cognitive load.
Recent advancements in prosthetic hands have brought sophisticated devices with multiple degrees of freedom. However, controlling these advanced prostheses intuitively remains a significant challenge for users. Traditional methods often rely on electromyography (EMG) signals from residual muscle activity, which can limit dexterity and increase the user’s cognitive load, sometimes leading to device abandonment.
A new research initiative, HannesImitation, is bridging this gap by applying imitation learning (IL) to prosthetic hand control. Imitation learning is a promising approach in robotics where systems learn complex tasks, like grasping, by observing demonstrations rather than relying on pre-programmed or manually labeled instructions. This method simplifies data collection and allows prosthetic devices to operate more naturally in diverse, unconstrained environments.
Introducing HannesImitationPolicy and Dataset
The core of this research is the HannesImitationPolicy, an imitation learning-based method designed to control the Hannes prosthetic hand. This policy enables the hand to grasp objects effectively in various unstructured settings. To train this policy, the researchers introduced the HannesImitationDataset, a unique collection of grasping demonstrations. This dataset includes scenarios such as grasping objects from a table, a shelf, and even human-to-prosthesis handovers, making it the first of its kind for imitation learning with prosthetic hands.
The Hannes prosthetic hand itself is a sophisticated device with three degrees of freedom: wrist flexion/extension, wrist pronation/supination, and fingers opening/closing. Crucially, it features a tiny RGB camera embedded in its palm, providing the visual feedback essential for the imitation learning approach. The HannesImitationDataset comprises 450 demonstrations using 15 different objects from the YCB dataset, ensuring a wide variety of shapes, masses, and colors. Each object was demonstrated 10 times in each scenario, with randomized initial poses and approach velocities to mimic real-world variability.
How the Policy Works
The HannesImitationPolicy leverages a technique called Diffusion Policy (DP), which is known for its ability to handle complex, multi-modal actions and maintain training stability. The policy takes input from the Hannes hand’s encoder measurements (for hand and wrist positions) and RGB images from the eye-in-hand camera. It then predicts a sequence of actions for hand opening/closing, wrist flexion/extension, and wrist pronation/supination, enabling the prosthetic hand to perform smooth and adaptive grasping behaviors.
Also Read:
- LightDP: Enabling Real-Time Robot Control on Mobile Devices
- Beyond Utility: How AI Can Be Designed to Empower Humans
Real-World Performance and Generalization
The HannesImitationPolicy was rigorously tested on the physical Hannes hand across all scenarios. It achieved an impressive overall success rate of 79.3% across 450 trials. Specifically, the success rates were 80.6% for Table Grasp, 68% for Shelf Grasp (which presented more challenging visual perspectives), and a remarkable 89.3% for Human-to-Hannes Handover. The high success in handovers highlights how the policy effectively complements natural human-subject interaction.
A key finding was the policy’s strong generalization capability. When tested on five YCB objects it had never encountered during training, the policy maintained a consistent overall success rate of 76%. This demonstrates its robustness in grasping new objects and adapting to different shapes, a testament to the diverse training data and the expressiveness of diffusion models.
The researchers also compared HannesImitationPolicy with a state-of-the-art visual servoing wrist controller. While the visual servoing method performed well in structured table and shelf scenarios (where it was specifically trained on those objects), HannesImitationPolicy outperformed it significantly in the challenging human-to-prosthesis handover scenario, where the visual servoing system struggled with object-subject distinction. This comparison underscores the effectiveness of imitation learning for enhancing generalization across varied and unstructured environments.
This research marks a significant step forward in prosthetic control, demonstrating the potential of imitation learning to enable more intuitive and dexterous control of prosthetic hands in real-world situations. Future work will explore more complex actions, broader comparisons, and user studies to assess real-world usability. You can find more details about this research on their project page: HannesImitation: Grasping with the Hannes Prosthetic Hand via Imitation Learning.
