TLDR: ColorBench is a novel graph-structured benchmark designed to evaluate mobile AI agents on complex, long-horizon tasks. It addresses the shortcomings of traditional offline static and online dynamic evaluation methods by simulating dynamic behaviors with static data. The benchmark supports multiple valid solutions, enables subtask completion rate statistics, and allows for atomic-level capability analysis. Experiments using ColorBench reveal limitations in existing models regarding task decomposition, memory, and reflection, providing insights and pathways for developing more robust mobile agents.
The world of mobile technology is constantly evolving, and with it, the need for sophisticated AI agents that can seamlessly interact with our devices. These mobile agents, powered by advanced multimodal large language models, promise to revolutionize how we use smartphones, making automation more accessible and user experiences smoother. However, evaluating the true capabilities of these agents has been a significant challenge.
Traditional evaluation methods fall into two main categories: offline static benchmarks and online dynamic testing. Offline static benchmarks, while easy to collect data for, are rigid. They only validate a single predefined “golden path” for a task, failing to account for the many valid ways a real-world task can be completed. This often leads to a discrepancy between an agent’s performance in a test and its actual ability on a device. Online dynamic testing, which involves evaluating agents in live environments, offers more authentic metrics but comes with its own set of problems. Real devices are unstable, prone to unexpected interruptions like ads or varying page load times, making consistent and reproducible testing difficult. Furthermore, assessing results is inefficient, often requiring manual verification, and there are security risks associated with agents logging into real accounts.
To address these critical issues, researchers from Shanghai Jiao Tong University and OPPO have introduced a novel solution: ColorBench. This innovative benchmark utilizes a graph-structured framework to evaluate mobile agents, aiming to bridge the gap between the stability of offline static evaluations and the flexibility of online dynamic testing. ColorBench models the finite states observed during real-device interactions, effectively achieving a static simulation of dynamic behaviors.
What Makes ColorBench Unique?
ColorBench is specifically designed for complex, long-horizon tasks, which are common in real-world mobile usage. These tasks often involve multiple steps, sometimes across different applications, and can have several correct ways to be completed. The benchmark includes 175 tasks, with 74 single-app and 101 cross-app scenarios, averaging over 13 steps each. Crucially, every task in ColorBench has at least two correct paths and several typical error paths, allowing for a quasi-dynamic interaction experience.
The graph-structured approach offers several key advantages:
- Multi-Solution Evaluation: Unlike rigid static benchmarks, ColorBench can assess agents based on multiple valid solutions for a single task, reflecting real-world flexibility.
- Enhanced Collaboration: It supports agents’ abilities to reflect on past actions and backtrack, preventing underestimation of their capabilities due to environmental constraints.
- Atomic Assessment: By setting milestones for subtasks within the graph, ColorBench enables stable, automated evaluation and fine-grained analysis of atomic task capabilities, helping diagnose specific weaknesses.
- Controllable Environment: It provides a secure, stable, and reproducible testing environment that is statically reliable yet supports rich interactions, avoiding security risks like unintended payments.
How ColorBench is Built
The construction of ColorBench involves a two-phase strategy: trajectory collection and graph merging. Trajectory collection gathers UI elements and interactions through both Breadth-First Search (BFS) for common, short-distance tasks and Depth-First Search (DFS) for complex, long-horizon tasks. The DFS method even incorporates “screenshot-based action completion” and “branch trajectory supplementation” to capture complete and realistic task flows, including potential error paths.
The graph merging phase then constructs the interactive evaluation environment. This involves using large language models for semantic coarse screening of screenshots and then fine-grained discrimination based on action transitions, ensuring that different screen states are accurately identified. A final manual verification step ensures data quality and completeness.
Also Read:
- SeqBench Uncovers Major Hurdles in Text-to-Video Narrative Generation
- DEEPPLANNER: A New Approach to Sharpen AI Agent Planning for Complex Research
Key Findings and Future Directions
Extensive experiments on ColorBench, involving both closed-source (like GPT-4o, Qwen-VL Max, GLM-4.5V) and open-source models, revealed significant insights. Researchers found that models often struggle with complex long-horizon tasks due to incomplete task decomposition, vague memory of historical information, and a lack of effective reflection on errors. For instance, some agents might complete a subtask like “share with contact” and incorrectly assume the entire task is done, or fail to retain critical information across different apps.
The study highlights that while basic GUI capabilities are essential, advanced competencies are crucial for complex tasks. These include the ability to comprehend and analyze complex task requirements, decompose long-horizon tasks into structured subtasks, actively memorize and recall critical information, and effectively reflect and self-correct. Ablation studies further demonstrated that while planning, reflection, and memory modules improve performance, their integration needs to be balanced to avoid instability and error accumulation.
In conclusion, ColorBench represents a significant step forward in evaluating mobile AI agents. By offering a stable yet dynamic testing environment, it provides a clearer picture of current agent limitations and offers concrete recommendations for developing more capable agents for the complex mobile world.
