TLDR: FOUNDER is a new AI framework that combines Foundation Models (FMs) for high-level understanding with World Models (WMs) for environmental dynamics. It learns to translate FM-understood tasks into WM-actionable goals, using temporal distance as a reward. This enables AI agents to solve diverse, open-ended tasks in simulated environments, outperforming prior methods, especially in complex or cross-domain scenarios, by fostering deeper task interpretation and avoiding reward hacking.
Artificial intelligence is making incredible strides, especially with powerful tools like Foundation Models (FMs) and World Models (WMs). FMs, trained on vast amounts of data, are excellent at understanding and generating things like text and images. WMs, on the other hand, are like internal simulators that help AI agents understand and predict how their environment works. While both are powerful, they have different strengths and challenges when it comes to creating AI that can interact with the real world.
FMs are great at general knowledge and understanding high-level concepts, but they struggle to directly control robots or agents in physical environments. Imagine an FM knowing what “open the microwave” means, but not how to translate that into specific robot movements. WMs are good at modeling the physics of an environment, allowing agents to “imagine” actions and their consequences, which makes learning more efficient. However, WMs typically need carefully designed “reward functions” to learn specific tasks, which can be very difficult, especially for tasks described in everyday language or videos.
This is where a new framework called FOUNDER comes in. Developed by researchers Yucen Wang, Rui Yu, Shenghua Wan, Le Gan, and De-Chuan Zhan, FOUNDER aims to combine the best of both worlds: the broad understanding of FMs with the dynamic modeling capabilities of WMs. The core idea is to “ground” the high-level knowledge from FMs into the actionable state space of WMs. This means translating what an FM understands about a task, like “run like a quadruped,” into a specific physical state or goal within the WM’s simulated world.
How FOUNDER Works
FOUNDER learns a special “mapping function” that acts like a translator. It takes the rich representations from an FM (for example, an embedding of a video showing someone running) and converts them into a state that the WM can understand and use. This allows the AI agent to infer its physical state in the simulator directly from external observations. Once the task is translated into a “goal state” within the WM, the agent can then learn how to achieve that goal through imagination, a process known as Goal-Conditioned Reinforcement Learning (GCRL).
A crucial innovation in FOUNDER is its reward system. Instead of relying on simple visual similarity, which can lead to agents mimicking appearances without truly completing a task (like waving arms near a light switch instead of turning it on), FOUNDER uses “temporal distance.” This means the agent is rewarded based on how close it is in terms of time steps to reaching the goal state. This encourages the agent to understand the deeper semantics of the task and perform meaningful actions, rather than just looking like it’s doing the task.
Also Read:
- Empowering Robots to Invent Their Own Tools with Vision Language Models
- How Perception, Memory, and Reasoning Modules Enhance AI in Games
Performance and Impact
The researchers put FOUNDER to the test on various complex tasks, including those from the DeepMind Control Suite and Franka Kitchen, and even in the challenging Minecraft environment. FOUNDER consistently showed superior performance, especially in scenarios with complex visual observations or when there was a significant “domain gap” – for instance, when the task video was from a different viewpoint or featured a different type of robot than the one being controlled. It excelled at understanding tasks specified by both text and videos.
Compared to previous methods like GenRL, FOUNDER demonstrated a deeper understanding of task semantics. While GenRL might focus on matching visual appearances step-by-step, FOUNDER’s approach of mapping to a goal state and using temporal distance allows for more adaptive and robust behavior learning. This was particularly evident in dynamic tasks like “running” or “walking,” where other methods sometimes failed due to “reward hacking” – the agent finding a way to get high rewards without actually performing the task correctly (e.g., running in place).
Even in the highly complex and open-ended Minecraft environment, FOUNDER performed remarkably well, often matching or surpassing oracle methods that rely on extensive pre-trained knowledge bases. This suggests that FOUNDER’s approach of learning a connection between FMs and WMs using limited environment data is highly efficient and adaptable.
FOUNDER represents a significant step forward in creating AI agents that can understand and solve a wide range of open-ended tasks in embodied environments. By effectively bridging the gap between high-level knowledge from Foundation Models and the low-level dynamics of World Models, it paves the way for more generalizable and robust AI systems. You can find the full research paper at this link.
