TLDR: InquireMobile is a new model that teaches VLM-based mobile agents to ask for human help in risky or ambiguous situations. It uses a two-stage training process and a new benchmark called InquireBench, significantly improving the agent’s ability to safely interact with users by proactively seeking confirmation.
Recent advancements in Vision-Language Models (VLMs) have empowered mobile agents to understand and interact with mobile environments based on human instructions. However, the current approach, where these agents operate fully autonomously, presents significant safety concerns. This is particularly true in high-stakes situations such as online payments, order placements, or the handling of sensitive personal data, where a misunderstanding by the agent could lead to severe, irreversible consequences.
Recognizing this critical gap, researchers have introduced a novel model called InquireMobile. This innovative system aims to develop an interactive mobile agent that actively seeks human confirmation at crucial decision points. By incorporating a human-in-the-loop safeguard, InquireMobile enhances the overall safety, transparency, and trustworthiness of mobile agent systems, moving beyond the blind reliance on purely autonomous model outputs.
To rigorously evaluate this new paradigm, the team also developed InquireBench, a comprehensive benchmark specifically designed to assess mobile agents’ capabilities in safe interaction and proactive inquiry with users. This benchmark covers 5 broad categories and 22 detailed sub-categories, including intent confirmation, privacy and security, and risk scenarios like file deletion or payment operations. Surprisingly, existing VLM-based agents demonstrated near-zero performance on InquireBench, underscoring the urgent need for solutions like InquireMobile.
How InquireMobile Works
InquireMobile employs a two-stage training strategy inspired by reinforcement learning. The first stage involves Supervised Fine-tuning (SFT), which equips the model with the ability to produce structured outputs and acquire fundamental graphical user interface (GUI) interaction skills. This initial phase ensures the agent can understand and respond in a coherent format.
The second stage utilizes Group Relative Policy Optimization (GRPO) training. This advanced reinforcement learning technique is crucial for enhancing the model’s reasoning and thinking capabilities. It helps the agent learn when it is truly necessary to ask for human assistance, preventing unnecessary interruptions while ensuring critical decisions are confirmed. A key component of InquireMobile is its interactive pre-action reasoning mechanism, where the agent proactively inquires from the user before executing potentially risky or ambiguous actions.
Also Read:
- PerPilot: A Framework for Personalized Mobile Agent Interactions
- Enhancing Vision-Language Model Safety with Principled Reasoning
Performance and Impact
Experiments conducted on InquireBench demonstrated InquireMobile’s significant effectiveness. The model achieved an impressive 46.8% improvement in inquiry success rate and the best overall success rate compared to existing baselines. This substantial performance gain validates the core idea that proactive user engagement is vital for the safe and effective deployment of mobile agents in real-world scenarios.
Furthermore, human satisfaction surveys revealed that InquireMobile, after its two-stage training, was the only model to achieve a satisfaction score above 2 on a five-point scale, indicating a better user experience. This highlights its ability to dynamically request human assistance appropriately, striking a balance between autonomy and necessary human intervention.
The researchers behind InquireMobile are committed to fostering further development in this field by open-sourcing all datasets, models, and evaluation codes. This work establishes a new paradigm for agent-human-environment interaction in mobile scenarios, ensuring that mobile agents can operate more safely and reliably in our daily lives. You can read the full research paper here.
