TARIPlay: A Test Framework for AR Applications based on Interactive Area Tracking in Playback Videos

2026-05-15

AI Summary

This study addresses the challenges of automated testing for augmented reality (AR) applications, which stem from their reliance on dynamic and non-deterministic real-world environments, particularly the difficulties in identifying interactive regions and determining appropriate interaction timing. To overcome these issues, this work proposes a novel automated testing approach based on replaying AR session videos. The method leverages computer vision to dynamically detect and track irregular, time-varying interactive regions, and integrates stability and visibility assessments to select valid test execution moments, thereby guiding the test engine to generate precise inputs. This approach represents the first technique capable of automatically tracking and filtering dynamic interactive regions from replay videos, effectively bridging environmental perception with test generation. Experiments on four open-source AR applications and nine replay videos demonstrate that, compared to the Monkey tool, the proposed method increases branch coverage from 41.98% to 55.8% and reliably evaluates the test suitability of replay videos.

Abstract

As Augmented Reality (AR) becomes more and more embedded in daily life, ensuring the quality, safety, and reliability of AR applications is increasingly important. However, AR apps present unique challenges for automated testing. Unlike static GUI layouts in traditional mobile apps, AR apps acquire their interaction interface from the surrounding environment, which is volatile and non-deterministic. Recent advancements like ARCore Playback and ARKit Replay allow developers to reuse real-world scenarios by recording and playing back enriched videos, enabling more feasible automated AR testing. However, using playback videos introduces two major challenges: test inputs must be timed precisely, and interactive areas in the video are dynamic, irregular, and difficult to identify. To address these challenges, we propose TARIPlay, a framework that analyzes playback videos to detect, track, and filter proper interactive areas over time for automated testing. In particular, TARIPlay identifies viable test opportunities based on criteria like stability and visibility, then feeds this information to an automated testing engine to simulate user interactions. We perform an experiment with four open-source AR apps and nine playback videos. Evaluation results show that TARIPlay significantly outperforms the existing tool Monkey in test coverage (55.8% over 41.98% on branch coverage) of AR-related code, and can also be used to assess the quality of playback videos for testing suitability.

Problem

• Augmented Reality
• automated testing
• interactive area tracking
• playback videos
• test input timing

Innovation

• interactive area tracking
• AR application testing
• playback video analysis
• automated testing
• test coverage