TLDR: Salesforce AI Research has introduced GTA1, a novel graphical user interface (GUI) agent that has achieved a new state-of-the-art in agentic human-computer interaction. GTA1 demonstrates superior performance, particularly on the OSWorld benchmark, where it surpassed OpenAI’s Computer-Using Agent (CUA) with a 45.2% task success rate. This advancement is attributed to its innovative test-time scaling strategy for planning and a reinforcement learning-based approach for precise visual grounding.
Salesforce AI Research has officially launched GTA1, a new graphical user interface (GUI) agent that is redefining the capabilities of autonomous interaction within real operating system environments, such as Linux. This innovative agent addresses two long-standing critical challenges in GUI agent development: the ambiguity inherent in task planning and the inaccuracy often found in grounding actions within complex interfaces.
GTA1 has quickly established itself as a leader among open-source models, setting a new performance record on the demanding OSWorld benchmark. It achieved an impressive 45.2% task success rate, significantly outperforming OpenAI’s Computer-Using Agent (CUA), which recorded 42.9%, and Claude 3.7, which stood at 28.0%. This achievement underscores GTA1’s robust decision-making and execution capabilities in diverse digital environments.
The agent’s superior performance extends beyond OSWorld. In the ScreenSpot-Pro benchmark for grounding accuracy, GTA1-7B scored 50.1%, surpassing models like UGround-72B (34.5%). Furthermore, on ScreenSpot-V2 for cross-platform grounding, GTA1-72B nearly matched top proprietary models with a 94.8% success rate. For Linux GUI grounding, specifically on OSWorld-G, GTA1-7B reached 67.7%, outperforming all previous open-source methods.
Salesforce AI attributes GTA1’s success to a modular two-stage framework, enhanced by a novel test-time planning diversity strategy. This strategy involves sampling multiple candidate actions per step and employing a multimodal judge to select the optimal option, ensuring robust decision-making without the need for future rollout. For grounding, GTA1 departs from traditional supervised learning, instead leveraging reinforcement learning with click-based rewards to directly predict valid interaction coordinates. This approach achieves state-of-the-art accuracy across complex, high-resolution GUI environments.
Notably, the research highlights that GTA1 achieves its state-of-the-art accuracy without the complexity or overhead of chain-of-thought style supervision. An ablation study even indicated that removing auxiliary signals, such as ‘thinking’ or IoU-based box rewards, actually improved grounding performance, particularly in static environments. This lean and effective architecture demonstrates that robust and accurate GUI agents can be built by focusing on core functionalities and precise RL-based grounding.
Also Read:
- GTA1: A New AI Agent for More Reliable GUI Automation
- Alibaba Unveils WebSailor: An Open-Source Web AI Agent Setting New Benchmarks
Salesforce AI Research has made the paper, codes, and various model sizes (7B, 32B, and 72B) publicly available, fostering further research and development in the field of agentic human-computer interaction.
