TLDR: A new method called Foundation modelGuided (FoG) skill discovery uses large AI models to teach robots desirable behaviors and avoid dangerous ones. Instead of just learning diverse skills randomly, FoG extracts a ‘score function’ from foundation models based on human preferences, which then re-weights the rewards for skill learning. This allows robots to learn skills that are not only varied but also safe and aligned with human intentions, even for complex or hard-to-define behaviors, in both simple and visual tasks.
Imagine a robot learning new skills, but instead of just trying everything, it learns what we actually want it to do – and more importantly, what it shouldn’t do. This is the core idea behind a new method called Foundation modelGuided (FoG) skill discovery, which aims to make robots learn diverse and desirable skills by incorporating human intentions.
Traditionally, robots learn skills through a process called reinforcement learning, which often requires carefully designed reward functions. Crafting these functions can be a huge effort for humans. Unsupervised skill discovery methods try to bypass this by encouraging robots to learn a wide variety of skills on their own. However, these methods often focus solely on maximizing diversity, which can lead to unexpected and even dangerous behaviors. For example, a robot cheetah might learn to roll around in all directions to maximize its skill set, when we’d really prefer it to run without flipping over or entering hazardous zones.
FoG addresses this challenge by leveraging the power of large AI models, known as foundation models (like ChatGPT, Claude, or CLIP). The method works by extracting a ‘score function’ from these models. This score function evaluates different states a robot might be in, assigning higher values to states that align with human intentions (desirable) and lower values to those that don’t (undesirable). These scores are then used to adjust the rewards given to the robot during its skill discovery process. By optimizing these re-weighted rewards, FoG guides the robot to eliminate undesirable actions, such as flipping, and to avoid dangerous areas.
The versatility of FoG is a key highlight. For tasks where the robot’s state is described by structured, numerical data (state-based tasks), foundation models can directly generate the score function. For tasks involving visual information (pixel-based tasks), FoG uses vision-language models like CLIP. These models generate embeddings (numerical representations) for both the robot’s current visual state and textual descriptions of desirable and undesirable intentions. The score function is then formed by comparing these embeddings, indicating how closely the current state matches a desired or undesired intention.
The researchers put FoG to the test against several existing methods in various environments. In state-based tasks like HalfCheetah and Ant, FoG successfully prevented a robot cheetah from flipping over and guided an Ant robot to avoid a specific ‘south’ area. The foundation models proved adept at identifying crucial aspects of the robot’s state, like the angle of a limb or its y-coordinate, to enforce these intentions.
In pixel-based tasks, where robots only see visual information, FoG continued to impress. It enabled a robot cheetah to run without flipping, significantly outperforming other methods that struggled to prevent such undesirable behaviors. When tasked with avoiding hazardous areas, FoG successfully biased the robot’s movement towards safe zones. Perhaps most interestingly, FoG demonstrated an ability to learn complex behaviors that are difficult for humans to define explicitly, such as making a humanoid robot appear ‘twisted’ or ‘stretched’. Human evaluators consistently preferred the skills learned by FoG in these subjective tasks.
An in-depth analysis also revealed important insights into FoG’s design. For instance, the parameter ‘alpha’, which controls how much undesirable states are penalized, showed that lower values generally lead to better performance by more strongly discouraging unwanted actions. The study also highlighted that directly using the noisy scores from foundation models as rewards for learning is not effective; instead, FoG’s approach of using them to re-weight existing skill discovery rewards is crucial for success.
Also Read:
- Enhancing Robot Dexterity: A New Framework for Generalizable Skill Learning
- Robots Learn Dexterous Skills from Everyday Human Videos
While FoG shows great promise in making skill discovery more aligned with human preferences, the researchers acknowledge that there’s always room for improvement. Future work might explore how to define score functions over sequences of states to capture more complex, process-based intentions, and how more advanced foundation models could further enhance performance. This research marks a significant step towards more autonomous and safer robot learning, where human intentions can seamlessly guide the discovery of useful and desirable skills. You can read the full research paper here: Guiding Skill Discovery with Foundation Models.
