APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents

2026-09-07

AI Summary

为解决移动GUI代理评估的真实性和可重复性问题，提出AppSim-Bench，通过可控的模拟应用支持确定性评估。

Abstract

Mobile GUI agents can execute tasks from natural-language instructions, but their evaluation remains difficult to make both realistic and reproducible. Existing benchmarks typically trade off these goals: simplified apps lack real-world mobile complexity, whereas live commercial apps introduce uncontrolled variation from recommendations, advertisements, accounts, and changing content. We propose AppSim-Bench, which addresses this trade-off through controllable simulated apps that preserve task-relevant interaction logic while supporting deterministic evaluation. Built through a coding-agent-assisted and human-verified workflow, it contains 557 tasks across 17 high-frequency Chinese and English apps. Its controllable backend data and outcome-based verification remove major sources of environmental stochasticity, enabling reproducible cross-model comparison. Evaluating 19 GUI agents, spanning general-purpose and GUI-specialized systems, we find that autonomous mobile execution remains far from solved. The best model completes only 50.27% of tasks, and 28.55% of tasks are not solved by any agent. Further analysis shows that failures concentrate in longer workflows, numerical reasoning tasks, and inefficient trajectories marked by high action overhead and budget exhaustion. Our project is available at https://github.com/Acrab-Agentic-Labs/AppSim.

Problem

• mobile GUI agents
• natural-language instructions
• reproducible evaluation
• real-world complexity
• uncontrolled variation

Innovation

• controllable simulated apps
• deterministic evaluation
• task-relevant interaction logic
• reproducible cross-model comparison
• outcome-based verification