ReInAgent: A Context-Aware GUI Agent Enabling Human-in-the-Loop Mobile Task Navigation

2025-10-09

AI Summary

Existing mobile GUI agents overemphasize autonomous execution while neglecting users’ active involvement in ambiguous, dynamic, and conflicting scenarios, resulting in poor task adaptability and outputs misaligned with user preferences. To address this, we propose ReInAgent—a human–AI collaborative, context-aware GUI agent framework. It introduces a novel tri-agent coordination architecture and a shared memory mechanism to enable proactive interaction, conflict-aware planning, and task reflection. Through slot-based information management, continuous contextual analysis, and multi-agent collaboration, ReInAgent achieves real-time adaptation to dynamic environments. Empirical evaluation on complex information-impoverished tasks shows that ReInAgent achieves a 25% higher success rate than Mobile-Agent-v2, with task outcomes significantly better aligned with users’ true intentions and preferences.

Technology Category

• Multiagent Systems: Adversarial Agents
• Humans and AI: Planning and Decision Support for Human-Machine Teams
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Mobile GUI agents exhibit substantial potential to facilitate and automate the execution of user tasks on mobile phones. However, exist mobile GUI agents predominantly privilege autonomous operation and neglect the necessity of active user engagement during task execution. This omission undermines their adaptability to information dilemmas including ambiguous, dynamically evolving, and conflicting task scenarios, leading to execution outcomes that deviate from genuine user requirements and preferences. To address these shortcomings, we propose ReInAgent, a context-aware multi-agent framework that leverages dynamic information management to enable human-in-the-loop mobile task navigation. ReInAgent integrates three specialized agents around a shared memory module: an information-managing agent for slot-based information management and proactive interaction with the user, a decision-making agent for conflict-aware planning, and a reflecting agent for task reflection and information consistency validation. Through continuous contextual information analysis and sustained user-agent collaboration, ReInAgent overcomes the limitation of existing approaches that rely on clear and static task assumptions. Consequently, it enables more adaptive and reliable mobile task navigation in complex, real-world scenarios. Experimental results demonstrate that ReInAgent effectively resolves information dilemmas and produces outcomes that are more closely aligned with genuine user preferences. Notably, on complex tasks involving information dilemmas, ReInAgent achieves a 25% higher success rate than Mobile-Agent-v2.

Problem

• Addresses mobile GUI agents' limited adaptability to ambiguous tasks
• Resolves information dilemmas in dynamically evolving mobile scenarios
• Improves alignment of automated outcomes with user preferences

Innovation

• Multi-agent framework with shared memory module
• Dynamic information management for human-in-the-loop interaction
• Continuous contextual analysis with user-agent collaboration