Mobile-Aptus: Confidence-Driven Proactive and Robust Interaction in MLLM-based Mobile-Using Agents

2026-05-27

AI Summary

This work addresses the challenges of over-execution and excessive human求助 commonly observed in multimodal large language model–driven mobile agents during task execution. To mitigate these issues, the authors propose a general confidence fusion framework that introduces confidence-driven decision-making into mobile agents for the first time. The framework operates through a two-stage mechanism: enhancing interactive capabilities and correcting confidence bias, enabling agents to autonomously and robustly choose between self-execution and requesting human intervention based on their internal confidence levels. The approach generates actions and associated confidence scores via supervised fine-tuning and calibrates confidence using semantic similarity retrieval and direct preference optimization. Experiments demonstrate that the method improves average task success rates by over 17% across four mobile agent benchmarks and by 26% in real-world dynamic environments, requiring only 0.64 human interventions per instruction.

Abstract

Recent advancements in multimodal large language models (MLLMs) have shown exceptional potential in enabling mobile-using agents to autonomously execute human instructions. However, fully automated agents often try to execute tasks even when they are unable to resolve them, leading to the problem of over-execution. Previous studies solve it by training a interactive mobile-using agents to let agents request human interaction when agents can not complete user instructions. However, we find that these interactive agents tend to exhibit over-soliciting behavior, relying excessively on human intervention. To mitigate both over-execution and over-soliciting, we propose a universal confidence integration framework that enables confidence-driven proactive and robust interaction in MLLM-based mobile-using agents. The framework consists of two stages: interaction capability empowerment and confidence bias correction. In the interaction capability empowerment stage, agents learn through supervised fine-tuning to output both actions and confidence scores. In the confidence bias correction stage, agents learn to output more accurate confidence scores by combining semantic similarity retrieval with direct preference optimization. Experimental results show Mobile-Aptus achieves state-of-the-art performance on the four popular mobile-using agent benchmarks: OS-Kairos, AITZ, Meta-GUI, and AndroidControl. Mobile-Aptus consistently outperforms all baselines in offline benchmarks, with an average improvement over 17\% in task success rate. In real-world dynamic experiments, Mobile-Aptus surpasses the baseline by 26% in task success rate with only 0.64 intervention steps per instruction. The codes are available at https://github.com/Wuzheng02/Mobile-Aptus.

Problem

• over-execution
• over-soliciting
• mobile-using agents
• multimodal large language models
• human-agent interaction

Innovation

• confidence-driven interaction
• multimodal large language models
• over-execution mitigation
• direct preference optimization
• mobile-using agents