AndroidReality: How Far Are Mobile Agents from the Real World?

2026-08-07

AI Summary

This work addresses the significant performance degradation of mobile agents in real-world environments caused by interface perturbations. The authors propose AndroidReality, a framework that introduces the first systematic three-axis taxonomy of mobile interface perturbations—encompassing state, transition, and action—and establishes a robustness evaluation benchmark with controllable perturbations built upon AndroidWorld. Their key contributions include uncovering substantial robustness gaps in existing agents under four common perturbation types and introducing a training-free, test-time introspective recovery mechanism (TTIR). Experimental results demonstrate that TTIR effectively mitigates task failures in both clean and perturbed settings, thereby validating the proposed benchmark’s utility for stress testing and vulnerability identification.

Abstract

Mobile agents have achieved promising results on clean online benchmarks such as AndroidWorld, yet their performance often degrades sharply in real-world deployment due to environmental variations and imperfect interface conditions. In this work, we introduce AndroidReality, a perturbation-based framework for evaluating and improving the robustness of mobile agents. Through a Markov Decision Process (MDP) perspective, we organize real-world interface variability into a principled taxonomy of perturbations along three axes: state, transition, and action. Guided by this taxonomy, we build a perturbed mobile benchmark on top of AndroidWorld with realistic and controllable perturbation injections, enabling systematic robustness evaluation of mobile agents. Our evaluation reveals substantial robustness gaps and four recurring error categories, motivating a simple training-free Test-Time Introspective Recovery (TTIR) mechanism that mitigates these failures on both perturbed and clean settings. Together, these results position robustness as a missing dimension in mobile agent evaluation and establish benchmark perturbation as an effective tool for both stress testing and surfacing latent weaknesses of mobile agents.

Problem

• mobile agents
• robustness
• real-world deployment
• environmental variations
• interface perturbations

Innovation

• AndroidReality
• robustness evaluation
• perturbation taxonomy
• Test-Time Introspective Recovery
• mobile agents