TLDR: The OS-M AP paper introduces a new benchmark for evaluating computer-using agents, featuring 416 tasks across 15 applications. It assesses agents along two dimensions: automation levels (L1-L5) and generalization scopes (S1-S3). Experiments show that even state-of-the-art agents achieve only 11.4% overall success, struggling significantly with higher-level tasks requiring perception, reasoning, and coordination, highlighting a large gap from human performance and the need for further research.
Computer-using agents, designed to understand user intentions and perform tasks across digital environments, are rapidly transforming how we interact with technology. Imagine a digital assistant like J.A.R.V.I.S. from Iron Man, seamlessly managing schedules, editing documents, or shopping online. While such capabilities are becoming a reality, a significant challenge remains: how do we accurately measure and improve these agents?
Current methods for evaluating these agents often fall short. They tend to treat tasks as simple collections, without considering the diverse nature of tasks or the specific abilities agents need. This makes it difficult to pinpoint exactly where agents succeed or fail, and how well they align with what real users actually need. To address this, researchers have introduced a new benchmark called OS-M AP.
Introducing OS-M AP: A Comprehensive Evaluation Framework
The OS-M AP benchmark is designed to provide a more structured and thorough way to assess computer-using agents. It features 416 realistic tasks spread across 15 different applications, focusing on everyday computer automation. What makes OS-M AP unique is its two-dimensional evaluation approach:
- Automation Levels: This dimension categorizes tasks based on the degree of autonomy an agent requires, from simple execution to complex proactive behaviors.
- Generalization Scope: This dimension measures an agent’s ability to handle tasks across various real-world user demands, from narrow, specific scenarios to broad, general use cases.
This framework allows for a detailed analysis of an agent’s capabilities, creating a performance-generalization matrix. This matrix helps researchers understand the strengths and limitations of different agents and guides future development towards more capable and versatile AI.
Understanding Automation Levels
The paper defines five levels of automation, inspired by classifications used for self-driving cars, but adapted for computer tasks:
- L1: Reactive Executor – The agent acts as a simple assistant, performing basic operations like clicks or typing exactly as instructed by the user. The user does all the planning.
- L2: Deterministic Planner – The user provides a goal, and the agent plans and executes actions in predictable environments. If something goes wrong, the user needs to intervene.
- L3: Adaptive Agent – Agents at this level can adjust their plans autonomously when unexpected events occur or the interface changes. Users define high-level goals but don’t need to monitor execution.
- L4: Global Conductor – These agents take full responsibility for breaking down complex goals, managing multiple applications, and using tools. Users only provide the initial goal and verify the final outcome.
- L5: Proactive Companion – This is the most advanced level, where the agent continuously monitors context, anticipates user needs, and initiates helpful actions without explicit instructions. OS-M AP currently does not include tasks for this level, leaving it for future research.
Generalization Across User Demands
To ensure the benchmark reflects real-world usage, tasks in OS-M AP are derived from a hierarchy of user demands. This hierarchy includes six main domains like work, study, and entertainment, further broken down into specific scenarios and tasks. Based on this, three scopes of generalization are defined:
- S1 (Narrow): Agents handle tasks within a single specific scenario (e.g., managing a calendar).
- S2 (Domain-Level): Agents succeed across multiple scenarios within a broader domain (e.g., document editing, emailing, and scheduling within the ‘work’ domain).
- S3 (General): Agents demonstrate S2-level performance across most or all six domains, acting as a versatile assistant for daily computer use.
Also Read:
- Navigating the AI Frontier: Large Language Models in Social Simulation
- New REST Framework Stress-Tests Large Reasoning Models for Multi-Problem Solving
Key Findings and Challenges
Experiments with State-of-the-Art agents on OS-M AP revealed that even the most advanced models, including those with Vision Language Model (VLM) backbones, struggle significantly. The overall success rate for these agents was only 11.4%, with near-zero performance on higher-level tasks (L3 and L4). This highlights a substantial gap between current AI capabilities and human performance in complex computer automation.
The research identified several common failure points:
- Poor instruction following: Agents sometimes fail to adhere to specified action spaces or output formats.
- Severe hallucination: Agents occasionally misinterpret the screen, leading to incorrect assumptions and actions.
- Struggles with non-textual elements: Even GUI-specific models find it hard to accurately locate visual elements without text.
- Planning issues: Agents can be distracted by similar but incorrect options or neglect specific task constraints.
- Lack of adaptability: Agents struggle to recover from unexpected situations or to use system-level interfaces when needed.
- Orchestration failures: For complex tasks involving multiple applications, agents often fail to break down tasks, track dependencies, or switch contexts effectively.
The study emphasizes that while open-source models fine-tuned on GUI-specific data show promise, and planning-grounding agents (which combine general models for planning with specialized models for precise control) perform better than general models alone, significant challenges remain in areas like adaptive reasoning and long-horizon planning.
The OS-M AP benchmark provides a crucial tool for the AI community, offering both qualitative and quantitative insights into the current state of computer-using agents. It serves as a clear roadmap for future research, guiding efforts towards building more robust, adaptable, and general-purpose AI assistants. You can find more details about this research paper here: OS-M AP Research Paper.
