TLDR: PerPilot is a new framework that enables VLM-based mobile agents to understand and execute personalized user instructions, which often contain ambiguous, user-specific context. It uses a Perception module to identify personalized elements and a Completion module that combines memory-based retrieval and reasoning-based exploration to find missing information. Tested with the new PerInstruct dataset, PerPilot significantly improves the success rates of existing mobile agents and learns to become more personalized with continued use, reducing the need for human intervention.
Mobile agents powered by advanced Vision Language Models (VLMs) hold immense promise for helping users with various tasks on their phones. These agents can understand instructions, recognize objects on screens, and navigate complex app interfaces to complete multi-step operations. However, a significant challenge has largely been overlooked: their inability to handle personalized instructions.
Personalized instructions are those that contain ambiguous, user-specific context. Think of commands like “Play my favorite song” or “Navigate to my home.” These instructions require the agent to understand individual preferences and routines, which current systems often struggle with because they are designed for clearly defined, explicit tasks like “Set an alarm for 7:30 am.”
Introducing PerInstruct: A New Benchmark for Personalization
To address this gap, researchers have introduced PerInstruct, the first benchmark specifically designed to evaluate how well VLM-based mobile agents can handle personalized tasks. PerInstruct includes 75 personalized instructions across 27 popular mobile applications, covering a wide range of everyday scenarios. These instructions are categorized into simple, normal, and hard levels based on the number of personalized elements and the complexity of the tasks involved. This dataset helps to systematically assess different degrees of personalization challenges.
PerPilot: Empowering Agents with Personalization
To enable mobile agents to autonomously understand and execute these personalized instructions, a new framework called PerPilot has been proposed. PerPilot is a plug-and-play system powered by Large Language Models (LLMs) that integrates with existing mobile agent systems. It works by identifying personalized elements in an instruction and then completing the missing information through two main approaches: memory-based retrieval and reasoning-based exploration.
When an agent receives an instruction, PerPilot’s Perception module first determines if it contains any user-specific personalized elements (e.g., “my home,” “my friend”). If it does, the Completion module takes over. It first checks a personalized memory database for previously stored information. For instance, if “mom” was previously linked to “Susan,” it retrieves that. If the information isn’t in memory, PerPilot then uses reasoning-based exploration. It intelligently figures out which mobile apps are most likely to contain the missing information (e.g., a shopping app for an address, a social app for a contact) and generates instructions for the agent to explore that app and retrieve the data. Once found, this new information is stored in memory for future use, and PerPilot then clarifies the original instruction with explicit details, allowing the mobile agent to complete the task.
Also Read:
- HumanoidVerse: A Robot That Understands and Rearranges Multiple Objects with Vision and Language
- WEBSIGHT: How Visual Perception is Advancing Web Automation
Significant Improvements and Learning Over Time
Experiments show that PerPilot dramatically improves the performance of existing mobile agent systems. When integrated with leading systems like UI-TARS, MobileAgent-v2, and AppAgent, PerPilot significantly boosted their task success rates. For example, UI-TARS saw its success rate jump from 12.0% to 68.0%. This demonstrates PerPilot’s effectiveness and its ability to adapt to various agent architectures.
The research also highlights that PerPilot’s personalization capabilities improve with continued use. Initially, the system might rely more on exploration and occasional human intervention to gather personalized information. However, as it interacts more frequently with users, it increasingly retrieves information directly from its memory, becoming smarter and more autonomous over time. This reduces the need for exploration and human assistance, making the agent more efficient and personalized.
A clear example of PerPilot in action involves a user asking to “search for recipe posts about my favorite takeout.” Without PerPilot, a standard agent would likely fail, searching directly for “my favorite takeout” and finding no relevant results. With PerPilot, the system first identifies “my favorite takeout” as a personalized element. If it’s not in memory, it might reason that a food delivery app like “Ele.me” could contain this information. It then explores “Ele.me” to find the user’s favorite takeout (e.g., “dumpling”). Once this is clarified, PerPilot updates the instruction to “search for recipe posts about dumpling” in the “Rednote” app, leading to successful task completion.
This innovative framework marks a significant step towards creating next-generation mobile agents that can truly understand and adapt to individual user preferences, moving beyond explicit commands to a more intuitive and personalized interaction experience. You can find the full research paper here: PerPilot: Personalizing VLM-based Mobile Agents via Memory and Exploration.
