SimulataR: Rapid Assisted Reality Prototyping using Design-Blended Videos

2025-01-27

AI Summary

Designing augmented reality (aR) systems requires extensive real-world testing, which is costly and time-consuming—especially due to dynamic environmental factors such as lighting conditions, background complexity, and user motion. To address this, we propose a desktop-based in-situ video compositing prototyping method: first-person real-world video is temporally and spatially aligned with interactive aR design prototypes and rendered in real time, enabling high-fidelity simulation without optical see-through head-mounted displays (OST-HMDs). Our approach introduces the first “design–real-video” spatiotemporal alignment paradigm, supporting reliable visual evaluation in indoor and outdoor environments under low-to-moderate illumination. A user study (n=12) confirms that the experience closely approximates that of physical OST-HMDs. Furthermore, two professional aR designers validated that the method reduces iterative design-validation cycles by over threefold, significantly improving both efficiency and accessibility of aR design evaluation.

Technology Category

• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Computer Vision: Computational Photography, Image & Video Synthesis
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Assisted Reality (aR) is a subfield of Augmented Reality (AR) that overlays information onto a user's immediate view via see-through head-mounted displays (OST-HMDs). This technology has proven to be effective and energy-efficient to support the user and information interaction for everyday wearable intelligent systems. The aR viewing experience, however, is affected by varying real-world backgrounds, lighting, and user movements, which makes designing for aR challenging. Designers have to test their designs in-situ across multiple real-world settings, which can be time-consuming and labor-intensive. We propose SimulataR, a cost-effective desktop-based approach for rapid aR prototyping using first-person-view context videos blended with design prototypes to simulate an aR experience. A field study involving 12 AR users comparing SimulataR to real OST-HMDs found that SimulataR can approximate the aR experience, particularly for indoors and in low-to-moderate lit outdoor environments. Case studies with two designers who used SimulataR in their design process demonstrates the potential of design-blended videos for rapid aR prototyping.

Problem

• Augmented Reality
• Environmental Changes
• Design Efficiency

Innovation

• SimulataR
• Augmented Reality Design
• Rapid Simulation