TLDR: The FingerTip 20K benchmark addresses limitations in current mobile AI agents by introducing two new tracks: proactive task suggestion and personalized task execution. It uses a large dataset of real-life mobile interactions from 91 users, including contextual information and historical data, to train and evaluate agents. Experiments show that existing models struggle with these tasks, but fine-tuning with FingerTip data significantly improves agents’ ability to anticipate user needs and adapt to individual preferences.
Mobile phones are an indispensable part of our daily lives, and the way we interact with them is constantly evolving. Recent advancements in large language models (LLMs) have led to the development of mobile Graphical User Interface (GUI) agents that can automate tasks on our phones. Imagine telling your phone, “Set an alarm for 7:30 for me,” and the agent understands and performs the necessary actions by observing the screen and generating clicks or types.
However, current mobile GUI agents have significant limitations. They are primarily reactive, meaning they only act when given explicit instructions. This passive approach can increase the effort users need to put in, as they must formulate detailed commands for every task. Moreover, these agents often fail to consider the user’s unique context, such as their past actions, preferences, or even the time and location, leading to a lack of personalized service.
Introducing FingerTip 20K: A New Benchmark for Smarter Mobile Agents
To address these challenges, researchers Qinglong Yang, Haoming Li, Haotian Zhao, Xiaokai Yan, Jingtao Ding, Fengli Xu, and Yong Li have introduced the FingerTip 20K benchmark. This innovative benchmark aims to push mobile GUI agents beyond simple instruction following, enabling them to be more proactive and personalized.
FingerTip 20K introduces two crucial new tracks:
-
Proactive Task Suggestion: In this track, the agent must anticipate what the user wants to do, even without explicit instructions. It achieves this by analyzing the current screen, the user’s historical intents, their profile, the current time, and the scenario (like being at home or office). The goal is for the agent to predict the user’s potential intent and offer helpful suggestions, such as “You might want to order food” if it’s lunchtime and the user has a history of doing so.
-
Personalized Task Execution: This track evaluates the agent’s ability to complete tasks while adapting to the user’s unique action preferences. While existing benchmarks focus on successful task completion, FingerTip 20K goes a step further by assessing if the agent’s sequence of actions aligns with how a specific user would typically perform that task. For example, some users might prefer navigating through menus, while others might use search bars more frequently.
Real-World Data for Real-World Intelligence
A key strength of FingerTip 20K is its unique data collection methodology. Unlike previous benchmarks that often rely on predefined tasks or simulated environments, FingerTip collected data from 91 real users over one to two months of their daily mobile phone usage. Users installed a special app that recorded their intents and actions, including screenshots, accessibility trees (detailed UI information), and action sequences. This approach ensures the data reflects genuine user behaviors and includes rich contextual information like user profiles, timestamps, and location categories.
The dataset is substantial, comprising 21,437 episodes across 506 different Android applications. This large-scale, user-oriented data is crucial for training agents that can understand and adapt to diverse real-world scenarios and individual preferences.
Also Read:
- UserBench: A New Benchmark for Evaluating How AI Agents Understand User Needs
- MapAgent: Enhancing Mobile Task Automation with Experience-Driven AI Planning
Experimental Insights and Future Potential
The researchers conducted experiments using various generalist large language models (like GPT-4.1 and Qwen-VL-Max) and specialized GUI-control agents. The results highlighted the significant challenges posed by the proactive and personalized tasks. Generalist models, while powerful, struggled with precise UI interactions and showed very low success rates in task execution. Even specialized GUI-control models, while performing better in execution, often failed to personalize their actions to match user preferences.
However, the study showed promising results when a smaller model (Qwen-2.5-VL-7B) was fine-tuned using the FingerTip data. This fine-tuned model demonstrated notable improvements in both proactive task suggestion and, crucially, in aligning its actions with user preferences. This indicates that with the right kind of user-oriented data, even smaller models can learn to anticipate needs and adapt their behavior, paving the way for truly intelligent and user-centric mobile AI agents.
FingerTip 20K represents a significant step forward in developing mobile agents that are not just reactive tools but proactive assistants, capable of understanding and adapting to the unique ways individuals interact with their devices. This work lays the groundwork for a future where our mobile phones can offer more seamless, intuitive, and personalized experiences.
