TLDR: DigiData is a new large-scale, high-quality dataset for training AI agents to control mobile devices, featuring diverse and complex tasks derived from thorough app exploration. It also introduces DigiData-Bench, a benchmark with dynamic and AI-powered evaluation methods, addressing the shortcomings of traditional metrics like step-accuracy. Experiments show DigiData significantly improves agent performance, especially when combined with Chain-of-Thought data, and highlights the potential of LLM judges for automated evaluation.
Imagine a future where your mobile device understands and executes complex tasks on your behalf, navigating apps and features with human-like intelligence. This vision is closer to reality thanks to new advancements in training and evaluating AI agents for mobile control. Researchers have introduced DigiData, a groundbreaking dataset, and DigiData-Bench, a robust evaluation framework, designed to accelerate the development of these general-purpose mobile control agents.
The Challenge of Training Smart Mobile Agents
Current AI agents capable of interacting with user interfaces hold immense potential to transform how we use digital devices. However, building truly general-purpose mobile control agents that can perform a wide range of complex tasks across various applications has been challenging. Existing datasets often fall short in terms of depth, diversity, and scale, typically deriving goals from unstructured interactions. This limits an agent’s ability to learn and leverage the full spectrum of app functionalities. Furthermore, traditional evaluation methods, such as simple step-accuracy, have proven insufficient for reliably assessing an agent’s performance on real-world, intricate tasks.
Introducing DigiData: A New Foundation for Mobile AI
DigiData is a large-scale, high-quality, diverse, and multi-modal dataset specifically designed to overcome these limitations. Unlike its predecessors, DigiData’s goals are meticulously crafted through a comprehensive exploration of app features. This unique approach ensures greater diversity and significantly higher goal complexity, pushing agents to master advanced functionalities that are often unfamiliar even to human users.
The data collection process for DigiData involves three key phases:
- Goal Curation: Trained human annotators exhaustively explore app features to curate a rich list of goals.
- Demonstrations Collection: Annotators create human demonstrations, recording sequences of actions to achieve these goals on physical or emulated Android devices.
- Trajectory Verification: A combination of AI-based (LLM judges) and human verification methods ensures the high quality of the collected trajectories, filtering out unsuccessful attempts.
DigiData stands out with 152,000 trajectories across 8,275 unique goals in 26 Android apps. It boasts superior data quality, with a significantly higher success rate of human-demonstrated trajectories compared to existing datasets. Moreover, DigiData is the first dataset of its scale to include multiple input modalities: screenshots, UI Tree descriptions (the underlying Android OS accessibility tree), and Chain-of-Thought (CoT) data generated by Llama 4. This CoT data provides detailed observations, action rationales, and expected UI changes, enhancing both agent performance and explainability.
DigiData-Bench: A Benchmark for Real-World Evaluation
Alongside the dataset, researchers also present DigiData-Bench, a benchmark for rigorously evaluating mobile control agents on complex, real-world tasks. It features 309 goals across 37 Android apps, categorized by app novelty (seen, familiar, novel) to test generalization capabilities.
DigiData-Bench supports two primary evaluation protocols:
- Human-assisted Dynamic Evaluation: Human workers set up initial app states, monitor agent actions, and judge task success, guided by precise protocols. This method offers the most general and accurate assessment.
- AI-assisted Dynamic Evaluation (DigiData-Bench-Auto): This automated end-to-end testing suite uses LLM judges to evaluate goal achievement, reducing the need for constant human involvement.
Experiments demonstrate that agents trained with DigiData, especially when incorporating Chain-of-Thought data, achieve significantly higher task success rates on DigiData-Bench compared to baselines, including powerful models like GPT4o and Qwen2.5VL. The research also highlights that dynamic evaluations, measuring task success rate, are far more reliable indicators of an agent’s capability than traditional step-accuracy metrics, which often fail to predict true performance.
Also Read:
- Enhancing AI Control Through Instruction Prioritization
- Unifying Software Engineering Evaluation for AI Coding Agents with SWE-Compass
The Path Forward
The introduction of DigiData and DigiData-Bench marks a significant step towards developing more intuitive and effective human-device interactions. While the current dataset is limited to 26 mobile apps, future work aims to expand its coverage, explore transfer learning to novel applications, and adapt these methods for computer-based interfaces. This research provides essential tools for training and evaluating general-purpose mobile control agents, paving the way for a future where AI seamlessly assists us in navigating our digital lives. You can find more details about this research paper here: DigiData: Training and Evaluating General-Purpose Mobile Control Agents.
