🤖 AI Summary
Traditional GUI testing—relying on source code or layout files—suffers from poor cross-platform compatibility and low robustness due to semantic and structural mismatches across platforms. To address this, this work conducts a systematic literature review of 271 papers and presents the first large-scale taxonomy of 92 vision-driven GUI testing approaches. We propose an end-to-end automated testing framework integrating computer vision techniques—including OCR, object detection, image similarity matching, UI element localization, and semantic understanding. Our analysis reveals paradigmatic advantages in real-world interface adaptation, cross-platform interoperability, and runtime robustness. We identify six core technical challenges and five emerging research directions (e.g., multimodal testing, AIGC-enhanced testing). Furthermore, we construct the first structured knowledge graph for vision-based GUI testing and deliver a practical, actionable research roadmap grounded in empirical findings and technological trends.
📝 Abstract
Graphical User Interface (GUI) has become one of the most significant parts of mobile applications (apps). It is a direct bridge between mobile apps and end users, which directly affects the end user's experience. Neglecting GUI quality can undermine the value and effectiveness of the entire mobile app solution. Significant research efforts have been devoted to GUI testing, one effective method to ensure mobile app quality. By conducting rigorous GUI testing, developers can ensure that the visual and interactive elements of the mobile apps not only meet functional requirements but also provide a seamless and user-friendly experience. However, traditional solutions, relying on the source code or layout files, have met challenges in both effectiveness and efficiency due to the gap between what is obtained and what app GUI actually presents. Vision-based mobile app GUI testing approaches emerged with the development of computer vision technologies and have achieved promising progress. In this survey paper, we provide a comprehensive investigation of the state-of-the-art techniques on 271 papers, among which 92 are vision-based studies. This survey covers different topics of GUI testing, like GUI test generation, GUI test record&replay, GUI testing framework, etc. Specifically, the research emphasis of this survey is placed mostly on how vision-based techniques outperform traditional solutions and have gradually taken a vital place in the GUI testing field. Based on the investigation of existing studies, we outline the challenges and opportunities of (vision-based) mobile app GUI testing and propose promising research directions with the combination of emerging techniques.
