Modeling Object Attention in Mobile AR for Intrinsic Cognitive Security

2025-10-27

AI Summary

This study addresses differential recall performance for physical versus virtual objects in mobile augmented reality (AR), uncovering a critical cognitive safety risk—systematic omission of mission-critical objects from user memory. Method: We propose recall probability as a proxy metric and develop an interpretable predictive framework grounded in formative constructs integrating object, scene, and user-state features. The framework employs partial least squares structural equation modeling (PLS-SEM) and is benchmarked against random forest and multilayer perceptron models. Key drivers—ambient illumination, physical-virtual object density ratio, and pose registration stability—are identified through model analysis. Contribution/Results: Across four empirical studies, the PLS-SEM model achieves top F1 scores in three, yielding individualized recall probability estimates to enable dynamic interface adaptation. This work introduces the first cognitive safety–oriented AR attention assurance mechanism, significantly improving memory reliability and system usability.

Technology Category

• Cognitive Modeling & Cognitive Systems: Applications
• Planning, Routing, and Scheduling: Activity and Plan Recognition
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements

Abstract

We study attention in mobile Augmented Reality (AR) using object recall as a proxy outcome. We observe that the ability to recall an object (physical or virtual) that was encountered in a mobile AR experience depends on many possible impact factors and attributes, with some objects being readily recalled while others are not, and some people recalling objects overall much better or worse than others. This opens up a potential cognitive attack in which adversaries might create conditions that make an AR user not recall certain potentially mission-critical objects. We explore whether a calibrated predictor of object recall can help shield against such cognitive attacks. We pool data from four mobile AR studies (with a total of 1,152 object recall probes) and fit a Partial Least Squares Structural Equation Model (PLS-SEM) with formative Object, Scene, and User State composites predicting recall, also benchmarking against Random Forest and multilayer perceptron classifiers. PLS-SEM attains the best F1 score in three of four studies. Additionally, path estimates identify lighting, augmentation density, AR registration stability, cognitive load, and AR familiarity as primary drivers. The model outputs per-object recall probabilities that can drive interface adjustments when predicted recall falls. Overall, PLS-SEM provides competitive accuracy with interpretable levers for design and evaluation in mobile AR.

Problem

• Modeling object recall in mobile AR for cognitive security
• Identifying factors affecting object recall in AR experiences
• Developing predictors to shield against cognitive attacks

Innovation

• Using PLS-SEM model to predict object recall
• Identifying key factors like lighting and cognitive load
• Providing interpretable probabilities for AR interface adjustments