Automated Testing of Prevalent 3D User Interactions in Virtual Reality Applications

2026-01-30

AI Summary

This work addresses the lack of effective automated testing methods for generating realistic 3D user interactions—such as grabbing and triggering—in virtual reality (VR) applications, as well as the absence of metrics to quantify interaction coverage. To bridge this gap, we propose XRintTest, which introduces the first systematic categorization of four prevalent VR interaction types, constructs an abstract interaction flow graph model, and integrates a dynamic exploration strategy with a controller input synthesis mechanism to enable efficient automated testing. Leveraging this framework, we develop XRBench3D, a benchmark comprising 456 distinct interactions that exposes critical design flaws and testability issues in current VR systems. Experimental results demonstrate that XRintTest achieves 93% interaction coverage on XRBench3D, outperforming random exploration by 6× in efficiency and 12× in effectiveness, while successfully uncovering various runtime anomalies and configuration defects.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Intelligent Robots: Human-Robot Interaction
• Computer Vision: 3D Computer Vision

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures

Abstract

Virtual Reality (VR) technologies offer immersive user experiences across various domains, but present unique testing challenges compared to traditional software. Existing VR testing approaches enable scene navigation and interaction activation, but lack the ability to automatically synthesise realistic 3D user inputs (e.g, grab and trigger actions via hand-held controllers). Automated testing that generates and executes such input remains an unresolved challenge. Furthermore, existing metrics fail to robustly capture diverse interaction coverage. This paper addresses these gaps through four key contributions. First, we empirically identify four prevalent interaction types in nine open-source VR projects: fire, manipulate, socket, and custom. Second, we introduce the Interaction Flow Graph, a novel abstraction that systematically models 3D user interactions by identifying targets, actions, and conditions. Third, we construct XRBench3D, a benchmark comprising ten VR scenes that encompass 456 distinct user interactions for evaluating VR interaction testing. Finally, we present XRintTest, an automated testing approach that leverages this graph for dynamic scene exploration and interaction execution. Evaluation on XRBench3D shows that XRintTest achieves great effectiveness, reaching 93% coverage of fire, manipulate and socket interactions across all scenes, and performing 12x more effectively and 6x more efficiently than random exploration. Moreover, XRintTest can detect runtime exceptions and non-exception interaction issues, including subtle configuration defects. In addition, the Interaction Flow Graph can reveal potential interaction design smells that may compromise intended functionality and hinder testing performance for VR applications.

Problem

• Virtual Reality
• Automated Testing
• 3D User Interactions
• Interaction Coverage
• User Input Synthesis

Innovation

• Interaction Flow Graph
• Automated VR Testing
• 3D User Interaction
• XRintTest
• XRBench3D