GUIWatcher: Automatically Detecting GUI Lags by Analyzing Mobile Application Screencasts

2025-02-06

AI Summary

Automated detection of GUI jank in mobile applications remains challenging due to reliance on intrusive instrumentation or source-code access. Method: This paper proposes the first purely vision-based, end-to-end framework that detects jitter frames, long-loading frames, and frozen frames directly from test screen recordings—without code modification or instrumentation. It integrates inter-frame optical flow motion analysis, temporal anomaly modeling, and multi-class imbalanced classification to enable fine-grained jank localization and severity prioritization. Contribution/Results: Evaluated on real-world production data, the method achieves an average precision of 0.91 and recall of 0.96. Deployed in a continuous monitoring system, it automatically generates structured debugging reports containing timestamps, frame sequences, and root-cause clues. This work pioneers systematic, non-intrusive visual analysis for GUI performance defect diagnosis, significantly improving both the efficiency and interpretability of mobile performance issue detection.

Technology Category

• Computer Vision: Motion & Tracking
• Search and Optimization: Evaluation and Analysis
• Machine Learning: Evaluation and Analysis

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements

Abstract

The Graphical User Interface (GUI) plays a central role in mobile applications, directly affecting usability and user satisfaction. Poor GUI performance, such as lag or unresponsiveness, can lead to negative user experience and decreased mobile application (app) ratings. In this paper, we present GUIWatcher, a framework designed to detect GUI lags by analyzing screencasts recorded during mobile app testing. GUIWatcher uses computer vision techniques to identify three types of lag-inducing frames (i.e., janky frames, long loading frames, and frozen frames) and prioritizes the most severe ones that significantly impact user experience. Our approach was evaluated using real-world mobile application tests, achieving high accuracy in detecting GUI lags in screencasts, with an average precision of 0.91 and recall of 0.96. The comprehensive bug reports generated from the lags detected by GUIWatcher help developers focus on the more critical issues and debug them efficiently. Additionally, GUIWatcher has been deployed in a real-world production environment, continuously monitoring app performance and successfully identifying critical GUI performance issues. By offering a practical solution for identifying and addressing GUI lags, GUIWatcher contributes to enhancing user satisfaction and the overall quality of mobile apps.

Problem

• Detects GUI lags in mobile apps
• Uses screencast analysis for lag detection
• Improves app performance and user satisfaction

Innovation

• Computer vision for lag detection
• Analyzes screencasts for GUI issues
• Generates comprehensive bug reports