TLDR: This research introduces GUI-Owl, a foundational GUI agent model that achieves new state-of-the-art performance in GUI automation across desktop and mobile environments. It unifies perception, grounding, reasoning, planning, and action execution. Built on Qwen2.5-VL, GUI-Owl leverages a self-evolving data production pipeline and scalable reinforcement learning. The paper also proposes Mobile-Agent-v3, a multi-agent framework that further enhances GUI-Owl’s capabilities through collaborative agents (Manager, Worker, Reflector, Notetaker), leading to superior performance in complex, long-horizon GUI tasks.
A new era of Graphical User Interface (GUI) automation is dawning with the introduction of GUI-Owl and the Mobile-Agent-v3 framework, developed by the Tongyi Lab at Alibaba Group. This groundbreaking research presents a foundational GUI agent model, GUI-Owl, which has achieved state-of-the-art performance across a wide array of GUI benchmarks, spanning both desktop and mobile environments. It addresses critical aspects of GUI automation, including grounding, question answering, planning, decision-making, and general procedural knowledge.
GUI-Owl is an end-to-end multimodal agent designed to be a cornerstone for GUI automation. Built upon the Qwen2.5-VL model, it undergoes extensive post-training using a vast and diverse collection of GUI interaction data. This training unifies perception, grounding, reasoning, planning, and action execution within a single policy network. The model demonstrates robust cross-platform interaction capabilities, adeptly handling multi-turn decision-making with clear intermediate reasoning. It can operate autonomously or serve as a specialized module within multi-agent frameworks.
One of the key innovations behind GUI-Owl is its large-scale environment infrastructure. This cloud-based virtual environment supports various operating systems, including Android, Ubuntu, macOS, and Windows. This infrastructure powers a Self-Evolving GUI Trajectory Production framework, which generates high-quality interaction data through sophisticated query generation and rigorous correctness judgment. This self-reinforcing improvement cycle continuously refines trajectories, reducing the need for manual annotation.
The framework also emphasizes diverse foundational agent capabilities. It incorporates essential UI data for grounding (localizing UI elements), planning (distilling procedural knowledge from successful trajectories and large language models), and action semantics (understanding how actions affect page states). Furthermore, GUI-Owl is enhanced with robust reasoning abilities through various data synthesis strategies, including offline hint-guided rejection sampling, distillation from multi-agent frameworks, and iterative online rejection sampling. These methods enrich the diversity of reasoning patterns and align them with real-world environmental dynamics.
A scalable reinforcement learning (RL) framework is another crucial component. This framework enables fully asynchronous training and better aligns the model’s decisions with real-world usage. It introduces Trajectory-aware Relative Policy Optimization (TRPO) for online environment RL, which effectively handles long, variable-length action sequences and sparse rewards by evaluating entire trajectories and using a replay buffer to stabilize training.
Building on GUI-Owl’s capabilities, the researchers propose Mobile-Agent-v3, a general-purpose GUI agent framework. This multi-agent system further boosts GUI-Owl’s performance, achieving new state-of-the-art results on benchmarks like AndroidWorld and OSWorld. Mobile-Agent-v3 coordinates four specialized agents: a Manager Agent for strategic planning, a Worker Agent for tactical execution, a Reflector Agent for self-correction and feedback, and a Notetaker Agent for maintaining persistent contextual memory. This collaborative architecture allows for robust, adaptive, and long-horizon GUI task automation.
The evaluation results are impressive. GUI-Owl-7B outperforms all state-of-the-art models of comparable size across various grounding and comprehensive GUI understanding benchmarks. GUI-Owl-32B demonstrates even more outstanding performance, surpassing proprietary models like GPT-4o and Claude 3.7 on several benchmarks. When integrated into Mobile-Agent-v3, it achieves even higher scores, showcasing its effectiveness as a fundamental agent for GUI automation.
Also Read:
- Adaptive Planning for LLM Agents with Grounded Memory
- Intelligent Configuration Optimization with Language-Guided Tuning
This research marks a significant step forward in creating intelligent agents that can interact with graphical user interfaces as seamlessly as humans. The open-sourcing of GUI-Owl and Mobile-Agent-v3 at https://github.com/X-PLUG/MobileAgent promises to accelerate further advancements in the field of GUI automation. For more in-depth technical details, you can refer to the full research paper.
