LLM-Guided Scenario-based GUI Testing

2025-06-05

AI Summary

Existing mobile GUI automation testing methods lack semantic understanding of business logic, resulting in inadequate coverage of critical test scenarios. To address this, we propose ScenGen, a novel framework driven by semantically annotated test scenarios derived from manual testing. ScenGen introduces the first five-agent collaborative architecture—comprising Observer, Decider, Executor, Supervisor, and Recorder—where “test scenario completion” serves as the primary exploration objective. The framework integrates large language model (LLM)-based semantic comprehension, GUI state awareness, semantic layout modeling, contextual memory, and runtime defect monitoring to ensure traceability and business consistency. Experimental evaluation across multiple mainstream mobile applications demonstrates that ScenGen significantly improves coverage of key business functionalities, achieving superior scenario completion rates and test accuracy compared to state-of-the-art approaches.

Technology Category

• Natural Language Processing: Generation
• Search and Optimization: Sampling/Simulation-based Search
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

The assurance of mobile app GUI is more and more significant. Automated GUI testing approaches of different strategies have been developed, while there are still huge gaps between the approaches and the app business logic, not taking the completion of specific testing scenarios as the exploration target, leading to the exploration missing of critical app functionalities. Learning from the manual testing, which takes testing scenarios with app business logic as the basic granularity, in this paper, we utilize the LLMs to understand the semantics presented in app GUI and how they are mapped in the testing context based on specific testing scenarios. Then, scenario-based GUI tests are generated with the guidance of multi-agent collaboration. Specifically, we propose ScenGen, a novel LLM-guided scenario-based GUI testing approach involving five agents to respectively take responsibilities of different phases of the manual testing process. The Observer perceives the app GUI state by extracting GUI widgets and forming GUI layouts, understanding the expressed semantics. Then the app GUI info is sent to the Decider to make decisions on target widgets based on the target testing scenarios. The decision-making process takes the completion of specific testing scenarios as the exploration target. The Executor then executes the demanding operations on the apps. The execution results are checked by the Supervisor on whether the generated tests are consistent with the completion target of the testing scenarios, ensuring the traceability of the test generation and execution. Furthermore, the corresponding GUI test operations are recorded to the context memory by Recorder as an important basis for further decision-making, meanwhile monitoring the runtime bug occurrences. ScenGen is evaluated and the results show that ScenGen can effectively generate scenario-based GUI tests guided by LLMs.

Problem

• Bridging gaps between automated GUI testing and app business logic
• Ensuring completion of specific testing scenarios as exploration target
• Generating traceable scenario-based GUI tests via multi-agent collaboration

Innovation

• LLM-guided multi-agent GUI testing
• Scenario-based test generation
• Semantic understanding of GUI widgets