TLDR: This paper surveys OS Agents, AI assistants powered by multimodal large language models (MLLMs) that operate computing devices like phones and computers. It covers their core components (environment, observation, action space) and capabilities (understanding, planning, grounding). The survey details how these agents are built using foundation models (architecture, training strategies like pre-training, supervised fine-tuning, reinforcement learning) and agent frameworks (perception, planning, memory, action). It also explains how their performance is evaluated through objective and subjective metrics across various benchmarks (mobile, desktop, web). Finally, the paper highlights key challenges such as safety, privacy, personalization, and self-evolution, pointing to future research directions in this rapidly advancing field.
The long-held aspiration of creating AI assistants as capable and versatile as J.A.R.V.I.S. from Iron Man is steadily moving closer to reality. Thanks to significant advancements in multimodal large language models (MLLMs), we are now seeing the emergence of sophisticated AI agents that can operate computing devices like computers and mobile phones. These are known as OS Agents, and they interact with operating systems and their interfaces, such as Graphical User Interfaces (GUIs), to automate a wide range of tasks.
This paper offers a comprehensive overview of these advanced OS Agents, starting with their fundamental building blocks. At their core, OS Agents consist of key components: the environment they operate within (like a desktop, mobile, or web platform), the observation space (the information they can perceive, such as screen images or textual data like HTML), and the action space (the interactions they can perform, like clicks, inputs, or navigation). Beyond these components, OS Agents possess essential capabilities including understanding complex tasks, planning sequences of actions, and grounding these plans into executable operations within the digital environment.
The construction of OS Agents involves two critical aspects. First, the development of domain-specific foundation models. These models are built upon existing large language models (LLMs) and MLLMs, sometimes with modified architectures to handle high-resolution images or integrate visual and textual information more effectively. Their training involves various strategies, including pre-training on vast datasets to enhance their understanding of GUIs, supervised fine-tuning to improve their planning and grounding abilities, and reinforcement learning, where agents learn optimal decision-making through trial and error, often combined with human demonstrations.
Second, effective agent frameworks are built around these foundation models. These frameworks typically include a perception module to gather and analyze environmental information, a planning module to break down complex tasks into manageable steps, a memory module to store information and accumulate experience, and an action module to execute specific operations. The memory component is particularly interesting, as it can be short-term (for immediate task context), long-term (for historical tasks and knowledge), or specific (for user preferences and application functions), allowing agents to learn and adapt over time.
Evaluating OS Agents is crucial to assess their performance. This involves both objective evaluations, which use standardized numerical metrics to measure accuracy and efficiency, and subjective evaluations, which involve human judgment to assess how well the agent’s output aligns with human expectations. Metrics can be granular, focusing on individual steps, or holistic, assessing overall task completion and resource utilization. Benchmarks are designed across different platforms—mobile, desktop, and web—and settings, ranging from static environments to dynamic, real-world scenarios, to test various tasks like GUI grounding, information processing, and complex agentic tasks.
Despite their rapid advancements, OS Agents face significant challenges. Safety and privacy are paramount concerns, as these agents interact with sensitive user data and can be vulnerable to adversarial attacks, such as indirect prompt injection or environmental distractions. Developing robust defense mechanisms is an ongoing area of research. Another key challenge is personalization and self-evolution. The dream of a truly personal AI assistant, like J.A.R.V.I.S., requires agents to continuously adapt to individual user preferences and learn from interactions over time. While memory mechanisms show promise, expanding memory modalities beyond text and managing them effectively remain open research questions.
Also Read:
- SEAgent: An AI Framework for Autonomous Software Proficiency
- Evaluating Trust in AI: A New Benchmark for Multimodal Model Confidence
This survey consolidates the current state of OS Agents research, providing valuable insights for both academic inquiry and industrial development. For more in-depth technical details, you can refer to the full research paper available at arXiv:2508.04482.
