TLDR: Finite Automata Extraction (FAE) is a novel neuro-symbolic approach that learns editable game logic as programs from gameplay videos. Unlike traditional neural network world models, FAE uses a custom domain-specific language called Retro Coder to generate interpretable and generalizable code, requiring less data and computation. Evaluated on Pac-Man and River Raid, FAE demonstrates comparable predictive performance to neural models while providing game developers with modifiable code, though it has limitations in handling complex sprite dynamics and game management functions.
World models, which are essentially compressed learned representations of an environment, have traditionally relied on neural networks. While powerful, these neural network models present challenges for game developers. They are difficult to modify, require vast amounts of data for training, and are computationally intensive, making them less accessible for many developers.
A new research paper introduces an innovative approach called Finite Automata Extraction (FAE) that aims to overcome these limitations. FAE learns a neuro-symbolic world model directly from gameplay videos, representing the learned environment dynamics as editable programs. This method promises a more precise model of the environment and generates more general code compared to previous approaches.
The core of FAE involves a novel domain-specific language (DSL) called Retro Coder. The pipeline begins by extracting frames from gameplay videos. A self-supervised model then learns a dictionary of sprites (the visual elements in the game) from these frames. Following this, FAE searches over Retro Coder to extract programs for each identified sprite, effectively learning their behavior within the game.
Retro Coder is designed to make sprite behaviors learnable and editable. Each program for a sprite consists of ‘if-then’ statements, allowing developers to easily understand and modify the logic. The language includes condition functions, such as checking if an entity exists in the map or if objects are neighboring (which signifies collisions), and action functions, like moving an object towards another entity or in a specific direction, or changing an object’s type.
The researchers evaluated FAE on two classic 2D game domains: Pac-Man and River Raid. These games were chosen for their distinct mechanics, providing a robust testbed. The evaluation compared FAE against existing neural world models like GameGAN and other DSL-based models such as Game Engine Learning.
In Pac-Man, FAE successfully learned programs for sprites like BLINKY (the red ghost), which learned to follow PACMAN and change into a GHOST when near PACMAN and a POWERPELLET. It also learned that POWERPELLETs effectively disappear when eaten by PACMAN. While some behaviors, like the randomness of GHOST and EYES sprites, were not fully captured, the system demonstrated its ability to learn significant game logic.
For River Raid, FAE learned that PELLETs move upwards and change to EMPTY at certain positions. It also captured collision behavior for FUEL. However, challenges arose with sprites that appeared rarely or varied in size, indicating areas for future improvement in the sprite representation and DSL.
The results showed that FAE achieved comparable or better performance in terms of prediction accuracy (measured by FID and Prediction Error) compared to GameGAN, which struggled with the limited training data. While Game Engine Learning had faster training times, it often failed to converge and produced less general code with many more conditions. FAE, despite a longer training time than Game Engine Learning, generated more concise and generalizable programs, making the learned code more practical for developers.
Also Read:
- AI Agents Learn to Hunt Bugs: A Smart Approach to Game Testing
- Automating Spreadsheet Formatting with Predictive AI
However, the current FAE approach has limitations. The sprite extraction can sometimes be inaccurate, especially when sprites are between grid cells or when dealing with sprite rotation or varying sizes. Retro Coder also currently lacks the ability to handle game manager functions (like sprite spawning), time-dependent behaviors, or randomized actions. Despite these limitations, FAE represents a significant step towards creating editable, low-data world models from gameplay video, offering a powerful tool for game development and analysis. You can read the full paper at arXiv.org.
