TLDR: A new framework, D2E (Desktop to Embodied AI), utilizes vast amounts of desktop interaction data, primarily from gaming, to pretrain AI models for robotics. It features the OWA Toolkit for efficient data collection and compression, a Generalist-IDM for pseudo-labeling YouTube videos to scale data, and VAPT for transferring these learned representations to physical robots. This method achieved high success rates on robot manipulation (96.6% on LIBERO) and navigation (83.3% on CANVAS), offering a scalable and cost-effective alternative to traditional physical data collection for embodied AI.
The field of embodied AI, which aims to create intelligent agents that can interact with the physical world, has long faced a significant hurdle: the immense cost and effort involved in collecting real-world data. Unlike large language models that thrive on internet-scale text, robots require specialized hardware, human operation, and complex annotation processes to gather interaction trajectories. This often results in small, fragmented datasets that limit the AI’s ability to generalize across diverse tasks.
The D2E Framework: A New Approach
A groundbreaking new framework, D2E (Desktop to Embodied AI), proposes a compelling solution: leveraging the rich, abundant, and cost-effective data generated from desktop environments, particularly gaming. Desktop interactions—involving screens, keyboards, and mice—offer a standardized, human-centric source of sensorimotor data. Crucially, these environments maintain the tight observation-action coupling essential for embodied learning, while abstracting away hardware-specific constraints. Gaming, in particular, provides complex patterns of navigation, object manipulation, and strategic planning that mirror many challenges faced by embodied AI.
OWA Toolkit: Efficient Data Capture
D2E introduces the Open-World Agents (OWA) Toolkit, a robust infrastructure for scalable desktop data capture. Traditional recording tools often lack the precise synchronization and efficiency needed for AI training. The OWA Toolkit’s ‘ocap recorder’ captures synchronized multimodal streams—screen, audio, keyboard, and mouse—with high temporal precision. Its innovative OWAMcap format achieves remarkable data compression, up to 152 times more efficient than existing formats, making large-scale data storage and processing feasible. Using OWA, the researchers collected 335 hours of human demonstrations across 31 diverse games and applications, laying a strong foundation for desktop-based pretraining.
Generalist-IDM: Scaling with Pseudo-Labeling
Beyond human demonstrations, D2E addresses the need for internet-scale data by introducing the Generalist Inverse Dynamics Model (Generalist-IDM). This model reformulates action prediction as timestamp-aware next-event prediction, allowing it to generalize across unseen games with minimal computational requirements. This generalization capability is key to automatically pseudo-labeling vast amounts of YouTube gameplay videos. By processing over 1,000 hours of unlabeled YouTube footage, the Generalist-IDM significantly expanded the training dataset without requiring additional human annotation, paving the way for truly internet-scale data collection for embodied AI.
VAPT: Transferring Desktop Knowledge to Robotics
The D2E framework culminates in Vision-Action PreTraining (VAPT), which demonstrates that representations learned from desktop interactions can meaningfully transfer to physical robotics. Models pretrained on the combined desktop corpus (human-collected and pseudo-labeled data, totaling over 1.3K hours) showed consistent improvements on standardized benchmarks. For robot manipulation, VAPT achieved an impressive 96.6% success rate on the LIBERO benchmark. For robot navigation, it reached 83.3% success on the CANVAS benchmark. These results validate that the sensorimotor patterns learned in digital environments possess sufficient invariance to enhance performance in physical embodied tasks.
Also Read:
- Robots Learn from Human Actions with EmbodiSwap
- ContextVLA: Enhancing Robot Dexterity with Efficient Temporal Understanding
A Practical Paradigm for Robotics
The D2E framework establishes desktop pretraining as a practical and scalable paradigm for robotics, significantly reducing the reliance on expensive physical data collection. By making all their work public—including the OWA toolkit, datasets, and VAPT-trained models—the researchers aim to empower the community to further explore and build upon this foundation. This approach democratizes embodied AI research, making large-scale vision-action pretraining accessible to a wider range of researchers and potentially accelerating the development of general-purpose intelligent agents.
For more in-depth information, you can read the full research paper here.
