TLDR: SEAgent is a novel self-evolving framework that enables computer use agents (CUAs) to autonomously learn and master unfamiliar software environments without human supervision. It utilizes a World State Model for precise step-level reward signals and a Curriculum Generator to create progressively challenging tasks. Through reinforcement learning from both successes and failures, and a specialist-to-generalist training strategy, SEAgent significantly improves agent performance on new software, demonstrating a path towards more versatile and independent AI.
In the rapidly evolving landscape of artificial intelligence, a significant challenge for computer use agents (CUAs) has been their reliance on extensive human-labeled data. While these agents, often powered by large vision-language models (LVLMs), have made impressive strides in interacting with computers, they frequently falter when confronted with new or specialized software, especially in scenarios where human annotations are scarce or non-existent.
Addressing this critical limitation, researchers have introduced SEAgent, an innovative self-evolving framework designed to empower CUAs to autonomously master unfamiliar software environments. This breakthrough system allows agents to learn through experiential learning, exploring new software, refining their skills through iterative trial-and-error, and progressively tackling tasks that evolve from simple to complex.
How SEAgent Works
SEAgent operates on a sophisticated pipeline comprising three main components: an Actor Model, a World State Model, and a Curriculum Generator.
The **Actor Model** is the agent’s policy, responsible for performing actions within the software environment based on the current state and task instructions.
The **World State Model (WSM)** is a fine-tuned Large Vision-Language Model that plays a crucial role in understanding the environment. It provides detailed descriptions of the software’s state and, more importantly, evaluates each step of the agent’s actions. This model can classify actions as either correct (contributing to success) or incorrect (leading to failure or redundant loops), providing precise, step-level reward signals for the agent’s learning process. Unlike previous models that only considered the final state, the WSM analyzes the entire sequence of screenshots, leading to more accurate judgments.
The **Curriculum Generator** is powered by a powerful Large Language Model. It automatically generates novel exploration tasks and maintains a continuously updated ‘software guidebook’ memory. This guidebook, enriched by the WSM’s feedback on state changes and task judgments, enables the Curriculum Generator to propose increasingly diverse and challenging tasks, fostering a curriculum learning paradigm.
Learning from Experience: Successes and Failures
SEAgent’s policy optimization is driven by experiential learning, drawing insights from both successful and failed actions. For correct actions, it employs a technique called Group Relative Policy Optimization (GRPO), which reinforces behaviors that lead to successful progression. Conversely, for incorrect actions, it uses ‘adversarial imitation’ to penalize and discourage failure-inducing behaviors, effectively teaching the agent to learn from its mistakes.
From Specialist to Generalist
A notable aspect of SEAgent is its unique ‘specialist-to-generalist’ training strategy. Initially, the framework trains software-specialized agents on individual environments, allowing each to become proficient in a specific application. These specialized insights are then distilled into a single generalist model through supervised fine-tuning. This generalist model is further refined across multiple software environments, ultimately achieving performance that surpasses even the combined capabilities of individual specialist agents.
Also Read:
- Autonomous AI Systems: Ushering in a New Era of Efficiency and Navigating Emerging Risks
- Foundations of Trustworthy AI Agents: Emphasizing Context, Constraints, and Control
Validation and Impact
The effectiveness of SEAgent was validated across five novel software environments within OS-World. The approach demonstrated a significant improvement in success rate, increasing from 11.3% to 34.5% over a competitive open-source CUA, UI-TARS. This substantial gain highlights SEAgent’s potential to develop more powerful and versatile computer-use agents without the need for human intervention.
This research marks a promising step towards a new generation of agents that can acquire superhuman capabilities by learning predominantly from experience, as envisioned by prominent AI researchers. While challenges remain, such as adapting to even more complex, hours-long workflows, SEAgent lays a robust foundation for future advancements in autonomous AI, potentially inspiring new research in gaming and real-world embodied environments. You can find more details about this research paper here.
