TLDR: A new research paper introduces the “Instruction-as-Reasoning” paradigm to enhance GUI grounding, a core capability for automated agents. The study found significant flaws in existing instruction datasets and untapped potential in leveraging instruction diversity. By training models with a two-stage SFT+RL framework, which treats instructions as dynamic analytical pathways, the resulting UI-Ins models achieve state-of-the-art performance on multiple benchmarks and demonstrate strong agentic potential in real-world tasks like AndroidWorld. The approach enables models to learn multi-perspective reasoning, select optimal pathways, and even develop emergent reasoning capabilities.
Automated agents that can interact with graphical user interfaces (GUIs) are a significant step towards achieving artificial general intelligence. A core capability for these agents is GUI grounding, which involves accurately mapping natural language instructions to the correct, actionable UI elements on a screen. Imagine telling your computer, “Click the red X to close this window,” and it understands exactly which ‘X’ to click. This is GUI grounding in action.
However, current approaches to GUI grounding often treat user instructions as static commands, overlooking the rich diversity and varying quality of how humans express their intentions. Researchers Liangyu Chen, Hanzhang Zhou, Chenglin Cai, Jianan Zhang, Panrong Tong, Quyu Kong, Xu Zhang, Chen Liu, Yuqi Liu, Wenxuan Wang, Yue Wang, Qin Jin, and Steven HOI from Renmin University of China, Tongyi Lab at Alibaba Group, and CUHK, have delved into this overlooked aspect. Their investigation revealed that a significant portion (23.3%) of instructions in existing grounding datasets contain flaws, introducing noise that can hinder model training. More importantly, they found that leveraging the diversity in how instructions are phrased could lead to substantial performance improvements, up to 76%.
To address these challenges, the team introduced a novel approach called the “Instruction-as-Reasoning” paradigm. Instead of viewing instructions as mere static inputs, this paradigm treats them as dynamic analytical pathways. Different ways of phrasing an instruction—such as describing a UI element by its appearance, function, location, or the user’s high-level intent—offer distinct perspectives for identifying the target element. An intelligent GUI agent, according to this paradigm, should not only understand a command but also actively select the most effective reasoning process to infer the user’s intent.
The development of this new paradigm involved a two-stage training framework. The first stage, Supervised Fine-Tuning (SFT), focused on synthesizing diverse instructions to teach the model multi-perspective reasoning. This means the model learned to generate an intermediate reasoning text, essentially a rewritten instruction from a specific perspective, before predicting the UI element’s coordinates. For example, if the original instruction was “Close the file manager window,” the model might learn to rephrase it as “Click the red X” (appearance) or “Close the file manager” (function).
The second stage employed Reinforcement Learning (RL) to optimize the selection and composition of these reasoning pathways. In this stage, the model was encouraged to “think” openly, without predefined perspectives, and learn which reasoning strategy was most effective for a given context. This iterative process, guided by rewards for accurate grounding, allowed the model to prioritize reasoning pathways that consistently led to correct predictions. Interestingly, the researchers observed that the model developed emergent capabilities, not only learning to select optimal pathways but also combining multiple perspectives and even formulating entirely novel reasoning perspectives not explicitly taught during training.
The models resulting from this framework, UI-Ins-7B and UI-Ins-32B, have achieved state-of-the-art results across five challenging grounding benchmarks. For instance, UI-Ins-32B scored 87.3% on UI-I2E-Bench, 57.0% on ScreenSpot-Pro, and 84.9% on MMBench-GUI L2, significantly outperforming previous methods. The benefits were particularly pronounced in more complex scenarios, where instructions required deeper understanding or inference. Furthermore, UI-Ins-7B demonstrated strong potential as an agent, achieving a 74.1% success rate on AndroidWorld when used as an executor with a GPT-5 planner, showcasing its practical utility in dynamic, real-world environments.
Also Read:
- Unlocking GUI Agent Potential: New Benchmark Reveals True Capabilities
- ColorAgent: Advancing OS Interaction with Intelligent and Personalized AI
The research also provided valuable insights into the nature of reasoning in GUI grounding. It confirmed that a structured “Instruction-as-Reasoning” approach consistently enhances performance, unlike unstructured “free-form reasoning” which can sometimes degrade it. The framework also effectively mitigates the common problem of “policy collapse” in SFT+RL setups by instilling diverse exploratory capabilities after the SFT stage. This foundational work highlights the critical role of instruction diversity and quality, and offers a robust framework for building more adaptable and intelligent GUI agents. For more details, you can refer to the full research paper.
