TLDR: FlashAdventure is a new benchmark of 34 Flash-based adventure games designed to evaluate GUI agents on completing entire story arcs, addressing the “observation-behavior gap.” It introduces CUA-as-a-Judge for automated evaluation and COAST, an agentic framework that uses long-term clue memory for better planning. Experiments show that while current AI agents struggle, COAST improves performance, though a significant gap with human ability persists.
Artificial intelligence (AI) agents are becoming increasingly capable of interacting with various digital environments, from browsing the web to using operating systems. Among these applications, video games offer a particularly rich and challenging testing ground for these AI systems. Adventure games, with their complex narratives, diverse interfaces, and need for logical reasoning, present unique hurdles for AI agents.
A new research paper introduces FlashAdventure, a benchmark designed to push the boundaries of what GUI (Graphical User Interface) agents can achieve in video games. Unlike previous benchmarks that often focus on short-term tasks or specific game mechanics, FlashAdventure evaluates agents on their ability to complete entire story arcs in 34 diverse Flash-based adventure games. This focus on full storylines highlights a critical challenge for AI: the “observation-behavior gap,” which refers to the time lag between when an agent observes a piece of information and when it needs to act upon it, often much later in the game.
The FlashAdventure benchmark includes a variety of classic adventure game subgenres, such as mystery/detective, hidden object, room escape, visual novel, and simulation games. These games were chosen because they emphasize reasoning over quick reactions, have clear progression milestones, and can be completed by humans within a reasonable timeframe (around one hour per game). This diversity ensures that AI agents are tested across a wide range of interaction styles and cognitive demands.
To facilitate reliable and automatic evaluation, the researchers developed CUA-as-a-Judge. This automated judge agent acts as an oracle, accessing predefined success milestones for each game and interacting with the game environment to verify if these milestones have been achieved. This innovative approach overcomes the limitations of manual assessment, which is common in many existing pixel/screenshot-based video game benchmarks.
In addition to the benchmark and evaluation system, the paper proposes COAST (Clue-Oriented Agent for Sequential Tasks), an agentic framework specifically designed to address the observation-behavior gap. COAST operates on a “Seek-Map-Solve” cycle:
Clue Seeking
The agent explores the game environment to collect potential clues, storing all gathered information in a long-term clue memory.
Clue-Observation Mapping
The agent analyzes its accumulated memory and gameplay history to identify promising connections between clues and past observations. Based on these connections, it generates plausible subtasks or hypotheses.
Also Read:
- Agentic Reinforcement Learning: Empowering LLMs as Autonomous Decision-Makers
- AppCopilot: Advancing Mobile AI Agents for Everyday Use
Problem Solving
The agent then executes actions to solve these proposed subtasks, updating its memory and filtering out resolved goals.
Experiments with FlashAdventure revealed that current state-of-the-art GUI agents struggle significantly with completing full story arcs. Common failure patterns include weak planning capabilities, poor visual perception in non-standard game layouts, and a lack of lateral thinking required for flexible problem-solving. COAST, by effectively managing clue memory and generating subtasks, showed improved performance in milestone completion and success rates compared to baseline agents, demonstrating its ability to bridge the observation-behavior gap and exhibit some degree of lateral thinking.
Despite these advancements, a substantial performance gap remains between the best-performing AI agents and human players. This indicates a need for continued research to enhance AI agents’ planning, perception, and reasoning abilities in complex, narrative-driven environments. The FlashAdventure benchmark and the COAST framework provide valuable tools and insights for future work in this exciting field.
For more in-depth technical details, you can read the full research paper here.
