TLDR: HuMam is a new end-to-end deep reinforcement learning framework for humanoid robots that uses a single-layer Mamba encoder to process robot states and external guidance. This approach significantly improves learning efficiency, training stability, and overall task performance compared to traditional methods. It also leads to more energy-efficient and smoother locomotion by reducing power consumption and joint torque peaks, making it the first humanoid RL controller to leverage Mamba for these benefits.
Humanoid robots have long captured our imagination, but making them walk with stability, efficiency, and foresight remains a significant challenge. Traditional control methods often struggle with the complexity of real-world environments and the intricate dynamics of a bipedal robot. This is where end-to-end reinforcement learning (RL) offers a promising path, allowing robots to learn complex behaviors directly from interaction.
A new research paper introduces HuMam, an innovative framework designed to enhance humanoid locomotion through a novel application of deep reinforcement learning. HuMam stands out by integrating a single-layer Mamba encoder, a type of state-space model, into its core architecture. This allows the robot to process diverse information – from its own joint positions and velocities to external guidance like planned footsteps and gait phase – in a highly efficient and stable manner.
The key innovation of HuMam lies in its Mamba encoder backbone. Unlike conventional feedforward networks or more computationally intensive sequence models like transformers, Mamba provides a lightweight yet powerful way to fuse heterogeneous inputs. It captures structured dependencies within the observation data without needing to accumulate temporal history, leading to smoother representations and reduced torque fluctuations in the robot’s movements. This is crucial for energy-efficient control and real-time deployment.
How HuMam Works
HuMam operates on a hierarchical control scheme. A high-level policy, trained with Proximal Policy Optimization (PPO), outputs desired joint positions at a rate of 40Hz. These targets are then translated into executable joint torques by a low-gain proportional-derivative (PD) controller running at a much faster 1000Hz. This combination ensures both learning stability and smooth, controlled actuation.
The learning process is guided by a carefully designed, concise six-term reward function. This reward system encourages a balance of critical locomotion aspects: maintaining good foot contact, ensuring smooth swing leg movements, precise foot placement, an upright body posture, a nominal body height, and upper-body steadiness. Importantly, this reward structure implicitly promotes energy saving, leading to more economical gaits.
Also Read:
- Enhancing Reinforcement Learning with Adaptive Demonstration Guidance
- Smart Navigation: A Hybrid AI Approach for Visually Impaired Mobility
Experimental Validation and Key Findings
The researchers conducted extensive experiments using the JVRC-1 humanoid robot in a simulated environment (mc-mujoco). HuMam was tested across various locomotion tasks, including forward, backward, curved, and lateral walking, as well as standing in place. The results were compared against a strong feedforward baseline, trained under identical conditions.
The findings were compelling:
- Improved Learning Efficiency and Stability: HuMam consistently achieved higher returns, learned faster, and exhibited significantly reduced variability across different training runs. It required fewer environment samples to reach performance thresholds, demonstrating superior sample efficiency.
- Enhanced Task Performance: The framework not only reached a higher peak performance but also maintained stronger outcomes over long training durations.
- Energy-Saving Control: HuMam showed remarkable improvements in energy efficiency. For instance, in standing, it reduced average power draw by 39%. In dynamic tasks like forward walking, it lowered energy cost from 673 J/m to 421 J/m. This is attributed to smoother joint actuation and fewer unnecessary corrective torques.
- Smoother Joint Torque Profiles: Analysis of joint torques revealed that HuMam reduced average joint torques by 9.6% in magnitude and 9.1% in peak load during forward walking, indicating less strenuous and more balanced actuation.
These results confirm that the Mamba encoder is an effective backbone for compact, end-to-end humanoid control, delivering tangible gains in efficiency, stability, and control economy. This work marks the first time Mamba has been adopted as the fusion backbone in an end-to-end humanoid reinforcement learning controller, showcasing its potential for future advancements in robotics.
For more in-depth details, you can read the full research paper here.
