Investigating Search Among Physical and Virtual Objects Under Different Lighting Conditions

2025-10-31

AI Summary

This study investigates the feasibility of augmented reality (AR) interaction in large-scale, real-world outdoor environments, focusing on how illumination conditions and cognitive load affect user behavior. Using Microsoft HoloLens 2, we conducted field experiments under natural daylight and nighttime conditions, integrating eye-tracking, head-pose estimation, walking trajectory logging, and an audio dual-task paradigm to systematically analyze search strategies, attention allocation, and memory performance for both physical and virtual targets. Results show that virtual objects are significantly more detectable at night, yet collision risk increases; users preferentially attend to virtual cues—even leveraging them to locate occluded physical targets; and high cognitive load induces strategic shifts in visual search, reducing recall accuracy for physical objects. This work presents the first quantitative characterization of the coupled effects of illumination and cognitive load on AR human–computer interaction in open outdoor settings, providing empirical foundations and behavioral modeling insights for designing robust, context-aware AR systems.

Technology Category

• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Computer Vision: Motion & Tracking

Application Category

• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

By situating computer-generated content in the physical world, mobile augmented reality (AR) can support many tasks that involve effective search and inspection of physical environments. Currently, there is limited information regarding the viability of using AR in realistic wide-area outdoor environments and how AR experiences affect human behavior in these environments. Here, we conducted a wide-area outdoor AR user study (n = 48) using a commercially available AR headset (Microsoft Hololens 2) to compare (1) user interactions with physical and virtual objects in the environment (2) the effects of different lighting conditions on user behavior and AR experience and (3) the impact of varying cognitive load on AR task performance. Participants engaged in a treasure hunt task where they searched for and classified virtual target items (green ``gems") in an augmented outdoor courtyard scene populated with physical and virtual objects. Cognitive load was manipulated so that in half the search trials users were required to monitor an audio stream and respond to specific target sounds. Walking paths, head orientation and eye gaze information were measured, and users were queried about their memory of encountered objects and provided feedback on the experience. Key findings included (1) Participants self-reported significantly lower comfort in the ambient natural light condition, with virtual objects more visible and participants more likely to walk into physical objects at night; (2) recall for physical objects was worse than for virtual objects, (3) participants discovered more gems hidden behind virtual objects than physical objects, implying higher attention on virtual objects and (4) dual-tasking modified search behavior. These results suggest there are important technical, perceptual and cognitive factors that must be considered.

Problem

• Evaluating AR search performance with physical and virtual objects
• Assessing lighting conditions' impact on user behavior in AR
• Investigating cognitive load effects on outdoor AR task performance

Innovation

• Used AR headset for outdoor user study
• Compared physical and virtual object interactions
• Measured cognitive load impact on AR performance