TLDR: A new reinforcement learning method, using a novel history encoder, enables humanoid robots to learn stable locomotion policies in the complex Gymnasium Humanoid-v4 environment even with significantly incomplete state information. This approach achieves performance comparable to or better than methods with full state access, demonstrating adaptability to robot property variations and offering a path towards more robust and sensor-efficient robotic control.
The long-standing dream of creating general-purpose humanoid robots capable of operating in diverse, real-world environments is a step closer to reality, thanks to a new breakthrough in reinforcement learning. While humanoid robots are designed to interact with spaces built for humans, controlling their complex movements, especially when they don’t have complete information about their surroundings, has been a significant challenge.
Traditional reinforcement learning methods often assume a ‘fully observed’ environment, meaning the robot has access to all necessary information about its state. However, in the real world, robots frequently operate under ‘partial observability’ due to incomplete sensors, measurement noise, or limited data transmission. This means the robot must make decisions based on incomplete or noisy information, much like humans do when navigating in low light or with limited senses.
Previous attempts to train humanoid robots under partial observability, particularly in high-dimensional tasks like the benchmark Gymnasium Humanoid-v4 environment, have struggled to achieve stable and effective policies. Existing approaches, often relying on recurrent neural networks (RNNs) like LSTM, have shown promise in simpler scenarios but faltered when faced with the complexity of humanoid locomotion.
A Novel Approach to Partial Observability
Researchers Wuhao Wang and Zhiyong Chen from The University of Newcastle have introduced a novel model-free reinforcement learning method that addresses this challenge. Their key innovation is a ‘history encoder’ that processes a fixed-length sequence of past observations in parallel. Unlike sequential memory-based approaches, this parallel processing allows the robot’s neural network to flexibly combine information across its observation window, effectively reconstructing essential contextual information from recent, incomplete observations.
This new method was rigorously tested in the Gymnasium Humanoid-v4 environment, a highly complex simulation with a 348-dimensional state space and a 17-dimensional action space. To simulate partial observability, the researchers selectively removed different components of the robot’s state information, such as velocity, mass/inertia, and force data, or combinations of these. This reduced the available information to as little as one-third of the original state dimensions.
Also Read:
- Optimizing Infrastructure Maintenance with Hierarchical AI Under Budget Constraints
- URBAN-SIM: Revolutionizing Autonomous Micromobility with Advanced Urban Simulation
Outperforming State-of-the-Art Baselines
The results were remarkable. The new method consistently outperformed two recent state-of-the-art memory-based approaches, RMF and ODERMF, which largely failed to learn effective policies in the humanoid task under partial observation. In contrast, the proposed approach achieved performance comparable to, and in some configurations even surpassed, policies trained with full state observations. This suggests that the history encoder is highly effective at inferring complete latent states from incomplete observation sequences.
The study also revealed insights into the importance of different sensory inputs. Velocity information was found to be the most critical for performance, while mass and inertia influenced how quickly the robot learned. Force information appeared to be the least essential. Notably, the learned policy also demonstrated impressive adaptability to variations in robot properties, such as changes in body part masses, further highlighting its robustness.
This research represents the first successful instance of stable training for humanoid policies with incomplete state information in the Gymnasium Humanoid-v4 environment. It paves the way for more robust and sensor-efficient robotic control systems, reducing reliance on extensive sensor setups and enhancing the adaptability of robots in real-world, unstructured environments. For more details, you can refer to the original research paper: Success in Humanoid Reinforcement Learning under Partial Observation.
