TLDR: ProBench is a new mobile benchmark for evaluating GUI (Graphical User Interface) agents, which are AI systems that interact with apps. Unlike traditional methods that only check the final screen, ProBench introduces “Process-related Tasks” that also assess if the agent followed critical intermediate steps. It uses a “Process Provider” to automatically capture these steps. Evaluations using ProBench show that current AI agents have significant limitations in understanding GUI elements, remembering past actions, and planning complex tasks, especially in social and lifestyle apps.
In the rapidly evolving world of artificial intelligence, Graphical User Interface (GUI) Agents are becoming increasingly important. These agents act as a bridge between human language goals and real-world devices, allowing us to interact with apps and systems more intuitively. However, evaluating how well these agents perform has been a significant challenge, often leading to an incomplete picture of their true capabilities.
Traditional methods for evaluating GUI agents typically focus only on the final outcome shown on a screen. For example, if an agent is asked to buy the cheapest wireless mouse, existing evaluations might only check if a wireless mouse appears in the purchase confirmation. This approach overlooks crucial intermediate steps, such as whether the agent actually sorted products by price from low to high before making a selection. If this step is missed, the agent might pick a random product, yet still be deemed successful by the old evaluation methods because a mouse was eventually “purchased.” This can lead to a false sense of success and mask real limitations.
Introducing ProBench: A New Standard for GUI Agent Evaluation
To address these shortcomings, researchers Leyang Yang, Ziwei Wang, Xiaoxuan Tang, Sheng Zhou, Dajun Chen, Wei Jiang, and Yong Li, along with their colleagues, have introduced a new benchmark called ProBench. This comprehensive mobile benchmark features over 200 challenging GUI tasks designed to cover a wide range of real-world scenarios across 34 popular Chinese and English online applications, including media, news, social, and shopping apps.
What makes ProBench unique is its dual approach to evaluation. It retains the traditional “State-related Task” evaluation, where the final screen state is sufficient for judgment. More importantly, it introduces “Process-related Task” evaluation. These tasks demand that the agent not only reaches the correct final state but also executes specific, critical intermediate operations. For instance, finding the highest-rated sushi restaurant in Tokyo and checking its full menu requires the agent to correctly locate Tokyo, sort restaurants by ratings, and select sushi restaurants—steps that aren’t always visible in the final screenshot alone.
To accurately capture these intermediate steps, ProBench features a “Process Provider.” This innovative component automatically supplies precise process information. It includes a “Structure Description Converter” that analyzes the app’s internal structure after each action, and an “MLLM-based Summarizer” that uses advanced AI models to compare screenshots before and after an action, generating a descriptive summary of what occurred. This ensures that the evaluation considers both the final outcome and the journey taken to get there.
Also Read:
- CrochetBench: Advancing AI’s Ability to Understand and Create Crochet Patterns
- Precision Training: Crafting Powerful GUI Agents with Filtered Data
Key Findings and Future Directions
The evaluation of advanced GUI agents on ProBench revealed significant limitations. Even the best-performing models struggled, successfully completing less than 50% of the tasks. These shortcomings were observed across various models, including large-scale generalist models and smaller, GUI-specific models. Agents particularly struggled with social and lifestyle applications, which often have frequently refreshing content, fragmented information display, and numerous visual distractions like pop-ups and icon-only buttons.
The research identified several universal problems:
- Insufficient Grounding Capability: Many models, especially proprietary ones like GPT-4o and Claude 4 Sonnet, found it difficult to accurately identify and locate GUI elements on the screen, leading to incorrect clicks or actions.
- Insensitivity to Historical Operations: Agents often failed to remember or utilize previous actions, leading to repetitive behaviors or an inability to recognize when a task step had already been completed. This was particularly evident in “Process-related Tasks.”
- Oversimplified Task Planning: When faced with complex, multi-step instructions, agents often resorted to simplistic strategies, such as typing the entire instruction into a search bar, rather than breaking down the task into logical sub-steps.
These findings highlight critical areas for improvement in GUI agents, emphasizing the need for better grounding, enhanced memory of historical actions, and more sophisticated task planning capabilities. ProBench aims to set a new standard for evaluating GUI agents, pushing the boundaries of what these intelligent systems can achieve in real-world mobile environments. You can read the full research paper for more details here.
