TLDR: StarDojo is a new benchmark using Stardew Valley to test AI agents, especially multimodal LLMs, on open-ended tasks involving both production (farming, crafting) and social interactions. Current MLLMs show significant limitations, achieving low success rates due to challenges in visual understanding, multimodal reasoning, and long-term planning, highlighting key areas for future AI research.
A new research paper introduces StarDojo, a groundbreaking benchmark designed to evaluate how well AI agents, particularly those powered by multimodal large language models (MLLMs), can navigate complex, open-ended environments that mimic real-world “production-living systems.” These systems involve both resource-generating activities and social interactions, a combination rarely tested simultaneously in existing AI benchmarks.
The StarDojo benchmark is built upon the popular simulation game Stardew Valley. This choice is strategic because Stardew Valley offers a rich, dynamic world where players engage in diverse activities like farming, crafting, exploration, combat, and social interactions. The game also realistically simulates time, stamina, weather patterns, and seasonal cycles, adding layers of complexity that challenge AI agents to adapt and plan effectively.
The researchers behind StarDojo have curated an extensive set of 1,000 tasks across these five key domains, with a smaller subset of 100 representative tasks for more efficient evaluation. These tasks range in difficulty from easy to hard, requiring agents to perform actions such as clearing farmland, planting crops, raising animals, mining for resources, crafting tools, trading with merchants, and even building relationships with non-player characters (NPCs).
One of the key innovations of StarDojo is its user-friendly interface. It provides an intuitive Python interface that interacts directly with Stardew Valley’s game engine, eliminating the need for manual keyboard and mouse controls or screen captures. This unified interface simplifies interaction and allows for parallel execution of multiple game instances, making it highly suitable for evaluating advanced MLLMs. The environment is also compatible with major operating systems like Ubuntu, macOS, and Windows.
Extensive evaluations of state-of-the-art MLLM agents, including models like GPT-4.1, Gemini 2.5 Pro, and Claude 3.7 Sonnet, revealed significant limitations. The best-performing model, GPT-4.1, achieved only a 12.7% success rate on the StarDojo-Lite task suite. Most successful completions were limited to easy tasks, with models struggling significantly with medium and hard tasks that demand longer action sequences and more complex reasoning. Challenges were particularly evident in visual understanding, multimodal reasoning, and low-level manipulation.
For instance, agents often failed to accurately locate and approach target objects in visual observations, hindering subsequent interactions. While textual observations provided supplementary information that helped with precise navigation, inconsistencies between visual and textual reasoning were observed. Agents also moved cautiously, often taking too many steps for distant targets, and struggled with long-term planning across multiple in-game days.
The research also included ablation studies, which showed the critical importance of both visual and textual inputs for agent performance. Removing either modality significantly impacted success rates. Furthermore, running the environment in “real-time” (without pausing the game during agent planning) drastically reduced performance, especially in tasks requiring timely reactions or prolonged action sequences, highlighting a crucial aspect often overlooked in other benchmarks.
StarDojo aims to facilitate further research into robust, open-ended agents capable of lifelong learning and long-term planning in complex production-living environments. While the benchmark currently focuses on early- and mid-game content and requires an official copy of Stardew Valley, it represents a significant step forward in evaluating AI agents in scenarios that closely mirror the multifaceted nature of real-world human society. You can find more details about this research at the project website: StarDojo Research Paper.
Also Read:
- Unlocking Deeper AI Understanding of Human Videos with HV-MMBench
- PlanQA: Uncovering How Language Models Navigate Spatial Challenges in Digital Floor Plans
The Stardew Valley Advantage
Stardew Valley provides an ideal backdrop for this benchmark due to its intricate systems. Each in-game day lasts about 14 minutes, requiring agents to manage time and energy effectively. Activities like farming, mining, foraging, fishing, and combat consume energy, which can be restored by sleeping or eating. The game also features four 28-day seasons and varying daily weather conditions, all impacting gameplay and requiring strategic adaptation.
Beyond production, social interaction is a core component. The game features 45 unique NPCs with distinct personalities and routines. Agents can build friendships, offer gifts, and participate in festivals and quests. This comprehensive economic and social system makes Stardew Valley a perfect testbed for evaluating AI agents in a holistic, real-world-like simulation.
