Adaptive 3D UI Placement in Mixed Reality Using Deep Reinforcement Learning

2024-05-11

CHI Extended Abstracts

AI Summary

To address the challenge of real-time adaptive 3D UI layout under dynamic user pose and environmental changes in mixed reality (MR), this paper introduces Proximal Policy Optimization (PPO)—the first application of deep reinforcement learning to continuous 3D UI placement. We propose a multimodal state encoding mechanism that fuses user pose with scene geometry, and design a task-oriented reward function to guide the policy network toward personalized, online-adaptive layout generation. Unlike conventional optimization methods relying on static assumptions, our approach overcomes their limitations by enabling real-time responsiveness in dynamic environments. Experiments demonstrate a 23% average improvement in task completion efficiency in mobile scenarios, layout response latency under 80 ms, and significant enhancements in content visibility and interaction accessibility.

Technology Category

• Machine Learning: Reinforcement Learning
• Search and Optimization: Mixed Discrete/Continuous Search
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Mixed Reality (MR) could assist users’ tasks by continuously integrating virtual content with their view of the physical environment. However, where and how to place these content to best support the users has been a challenging problem due to the dynamic nature of MR experiences. In contrast to prior work that investigates optimization-based methods, we are exploring how reinforcement learning (RL) could assist with continuous 3D content placement that is aware of users’ poses and their surrounding environments. Through an initial exploration and preliminary evaluation, our results demonstrate the potential of RL to position content that maximizes the reward for users on the go. We further identify future directions for research that could harness the power of RL for personalized and optimized UI and content placement in MR.

Problem

• Optimizing 3D UI placement in Mixed Reality dynamically
• Using reinforcement learning for user-aware content positioning
• Enhancing MR experiences with adaptive and personalized interfaces

Innovation

• Uses deep reinforcement learning for 3D UI placement
• Adapts to users' poses and environments dynamically
• Maximizes user reward through continuous content positioning