TLDR: VideoAgentTrek is a new pipeline that automatically extracts training data for computer-use AI agents from unlabeled public screen-recorded videos, like YouTube tutorials. It uses a module called VIDEO2ACTION to detect GUI actions and extract parameters like clicks and typed text. This method generated 1.52 million interaction steps from 39,000 videos, significantly improving agent performance on benchmarks like OSWorld-Verified (70% relative improvement) and AgentNetBench, offering a scalable and cost-effective alternative to manual data annotation.
Training artificial intelligence agents to use computers like humans do, performing tasks such as clicking buttons, typing text, and navigating interfaces, is a significant challenge in AI. A major bottleneck in developing these computer-use agents is the immense amount of graphical user interface (GUI) interaction data required. Traditionally, creating such datasets involves costly and time-consuming manual annotation of action trajectories, which makes it impractical to achieve the scale needed for robust AI performance across diverse applications and operating systems.
A new research paper introduces a groundbreaking solution called VideoAgentTrek, a scalable pipeline designed to automatically mine training data from publicly available screen-recorded videos. This innovative approach eliminates the need for manual annotation, leveraging the vast resource of online tutorial videos where humans implicitly demonstrate computer use.
The Core Innovation: VIDEO2ACTION
The heart of VideoAgentTrek is an inverse dynamics module (IDM) named VIDEO2ACTION. This module addresses the key challenge that raw videos contain implicit demonstrations but lack explicit action labels. VIDEO2ACTION comprises two main components:
- A video grounding model that accurately detects and localizes GUI actions within videos, identifying their precise temporal boundaries and context.
- An action-content recognizer that extracts structured parameters, such as exact click coordinates (x,y) and the specific text typed, with high fidelity.
By combining these components, VIDEO2ACTION can effectively convert passive video recordings into active training data, generating complete trajectories that include screenshots, action types, and their corresponding parameters.
Data Collection and Training
The VideoAgentTrek pipeline was applied to a massive dataset of 39,000 YouTube tutorial videos. From this corpus, the system automatically generated an impressive 1.52 million interaction steps. This data is not only extensive but also highly diverse, spanning hundreds of applications across Windows, macOS, and various web platforms, capturing interaction patterns that would be nearly impossible to annotate manually.
The researchers utilized this automatically mined data through a two-stage training recipe: continued pretraining on the extracted trajectories, followed by supervised fine-tuning on a curated dataset. This hybrid approach allows the agents to learn fundamental GUI interaction patterns from the broad coverage of videos, while supervised fine-tuning refines their task-specific performance.
Significant Performance Improvements
The results demonstrate the effectiveness of VideoAgentTrek:
- On OSWorld-Verified, a benchmark for online computer-use agents, the approach improved task success rates from a 9.3% baseline (SFT-only) to 15.8%, representing a substantial 70% relative improvement.
- On AgentNetBench, an offline benchmark, step accuracy increased from 64.1% to 69.3%.
These gains were particularly noticeable in online environments, highlighting the agent’s improved robustness to visual variations. The study also showed that video pretraining enables effective test-time scaling, meaning agents trained with VideoAgentTrek can utilize additional exploration opportunities to further improve performance on longer, more complex tasks.
Also Read:
- FineVision: A Unified Data Resource for Vision-Language Models
- Unlocking Deeper Video Reasoning Through Entropy Control
Open-Source Tools and Future Impact
The research team plans to release SCREENFILTER, a tool for efficient GUI filtering, and VIDEO2ACTION, for action extraction, as open-source tools. This will enable the broader research community to leverage this abundant resource for advancing GUI automation research.
VideoAgentTrek establishes that passive internet videos can be transformed into high-quality supervision for computer-use agents, offering a scalable and cost-effective alternative to expensive manual annotation. This work paves the way for more robust and capable AI agents that can interact with computers in increasingly human-like ways. You can read the full research paper for more details. Read the full research paper here.
