TLDR: Hi-Agent is a new AI system for controlling mobile devices, using a hierarchical approach with a high-level reasoning model and a low-level action model that learn together. It breaks down complex tasks into simpler subgoals and uses a “foresight advantage function” to guide its learning, leading to state-of-the-art performance and better generalization on various mobile control benchmarks.
The quest to build intelligent agents that can autonomously operate mobile devices has seen significant advancements, particularly with the rise of Vision-Language Models (VLMs). However, many existing methods struggle with complex tasks and unfamiliar user interfaces because they rely on direct mappings from screen states to actions, lacking structured reasoning and planning capabilities.
A new research paper introduces Hi-Agent, a trainable hierarchical vision-language agent designed specifically for mobile device control. This innovative system features a high-level reasoning model and a low-level action model that are optimized together, allowing for more robust and adaptable performance.
A Hierarchical Approach to Mobile Control
Hi-Agent tackles the limitations of previous models by adopting a hierarchical architecture. Instead of a single model trying to do everything, Hi-Agent separates the complex problem into two distinct but interconnected parts:
- High-level reasoning model (Ï€h): This model is responsible for understanding the overall task and breaking it down into a sequence of semantic subgoals. For example, if the task is “Send a message to Alice,” the reasoning model might generate subgoals like “Open Messenger,” “Search Alice,” and “Compose message.”
- Low-level action model (πℓ): This model takes the semantic subgoals from the high-level model and translates them into concrete, atomic UI actions, such as clicking a specific icon, typing text, or performing a swipe gesture.
This decomposition mirrors how humans approach complex tasks, first planning at a high level and then executing specific steps. Crucially, both of Hi-Agent’s models are trainable and co-adapted, meaning they learn and improve together. This is a significant departure from earlier hierarchical agents where the high-level planner often remained frozen, limiting true end-to-end learning.
Efficient Training with Foresight
To enable efficient training of this hierarchical structure, the researchers reformulated multi-step decision-making into a sequence of single-step subgoals. This significantly reduces the computational complexity associated with long-horizon tasks. They also introduced a novel “foresight advantage function.” This function uses feedback from the low-level model’s execution to guide the optimization of the high-level reasoning model. This mechanism helps the high-level model learn what constitutes an effective subgoal, not just for immediate execution but also for long-term task success, without needing a separate critic model.
State-of-the-Art Performance and Generalization
Hi-Agent has demonstrated impressive results, achieving a new state-of-the-art 87.9% task success rate on the challenging Android-in-the-Wild (AitW) benchmark. This significantly outperforms prior methods across various paradigms, including prompt-based (e.g., AppAgent: 17.7%), supervised (e.g., Filtered BC: 54.5%), and reinforcement learning-based (e.g., DigiRL: 71.9%) approaches.
Beyond raw performance, Hi-Agent also shows strong generalization capabilities. When tested with perturbed UI layouts (e.g., starting from an all-apps view instead of the home screen), Hi-Agent maintained high effectiveness, while other methods saw sharp drops in performance. This robustness highlights its ability to adapt to unseen UI configurations rather than relying on memorized coordinates. Furthermore, its low-level action model demonstrated competitive zero-shot generalization on the ScreenSpot-v2 benchmark, indicating it learns robust visual representations.
Also Read:
- DEEPPLANNER: A New Approach to Sharpen AI Agent Planning for Complex Research
- ColorBench: A New Graph-Based Benchmark for Advanced Mobile AI
Scalability for Complex Tasks
The research also explored Hi-Agent’s scalability with larger backbone models on the more complex AndroidWorld benchmark, which demands stronger reasoning and fine-grained control. The system scaled effectively, with a configuration using a 72B reasoning model and a 7B action model achieving a 56.5% success rate, significantly outperforming other baselines. This demonstrates Hi-Agent’s adaptability to high-complexity mobile control scenarios.
In conclusion, Hi-Agent represents a significant step forward in building intelligent agents for mobile device control. By decoupling high-level reasoning from low-level action execution and jointly optimizing these components with a foresight-guided training framework, it offers a powerful, robust, and scalable solution for automating mobile device operations. You can read the full research paper here: HI-AGENT: HIERARCHICAL VISION-LANGUAGE AGENTS FOR MOBILE DEVICE CONTROL.
