GUITestScape: Towards Open-set Evaluation on Exploratory GUI Testing

2026-05-28

AI Summary

This work addresses the limitations of existing GUI testing evaluations, which are confined to predefined interaction bugs, overlook visual display issues, and rely on single end-state judgments that fail to distinguish failure modes. To overcome these shortcomings, we propose GUIJudge—the first open-set evaluation framework that jointly encompasses both interaction and display defects. We construct an interactive benchmark comprising 61 Android applications and 508 real-world bugs. Our approach enables process-aware diagnostic capability assessment through trajectory decomposition, integrating a multimodal large language model agent, capability-level verifiers, and a training-free detection enhancement mechanism. Experiments demonstrate that GUIJudge substantially outperforms existing baselines, revealing defect detection as the primary bottleneck for current models, and show that incorporating its verifiers effectively boosts agent performance.

Abstract

Exploratory GUI testing is a particularly demanding setting for MLLM agents: without predefined test scripts, an agent must autonomously navigate an application and discover defects through its own interaction. However, current evaluation falls short on two fronts. First, existing benchmarks focus almost exclusively on interaction defects, leaving display defects outside the evaluation frame. Second, evaluation protocols are bound to predefined defect annotations, collapsing the testing process into a single end-state judgment that conflates qualitatively distinct failure modes. To address these challenges, we present GUITestScape, an interactive benchmark covering 61 real-world Android applications and 508 preset defects spanning interaction and display types, and introduce GUIJudge, an open-set evaluator that decomposes an agent's testing trajectory into independently diagnosable capabilities. Experimental results demonstrate that GUIJudge achieves reliable process-aware evaluation beyond predefined annotations, substantially outperforming all baselines. Benchmarking on GUITestScape further reveals that detection remains the critical bottleneck for existing models across both defect types, and that integrating GUIJudge's verifiers into existing agents significantly boosts their detection performance without retraining.

Problem

• exploratory GUI testing
• interaction defects
• display defects
• evaluation benchmark
• open-set evaluation

Innovation

• open-set evaluation
• exploratory GUI testing
• process-aware assessment
• defect detection
• MLLM agents