Problem
Research questions and friction points this paper is trying to address.
AR interaction
proxy-based interfaces
lightweight AR glasses
real-world object interaction
device-specific design
Generative Proxy: Synthesizing Proxy-Based Interfaces for Real-World Interaction Across AR Glasses
🤖 AI Summary
This work addresses the challenge of directly manipulating real-world objects with lightweight AR glasses, which is hindered by monocular vision, narrow field-of-view, and discrete input modalities—particularly in distant, occluded, or cluttered environments. The authors propose an automatic proxy interface generation method that formulates interface synthesis as a constrained optimization problem, integrating three key constraints: scene semantics, user intent, and device capabilities. Candidate interfaces are ranked using semantic and operational distance metrics. This approach enables, for the first time, cross-device, task-adaptive automatic synthesis of proxy interfaces, embedding principles of direct manipulation into both generation and evaluation. Experiments demonstrate that the method consistently produces usable interfaces across diverse scenarios and configurations, with expert evaluations confirming its effectiveness and practicality, thereby advancing proxy abstraction as a mainstream interaction paradigm in augmented reality.
📝 Abstract
Interacting with real-world objects in AR is difficult, especially when targets are distant, cluttered, or occluded. These challenges are amplified on emerging lightweight AR glasses, which often lack binocular or large field of view on display, but also continuous inputs, such as hand or eye tracking. Proxy-based interfaces offer an alternative by allowing users to interact with virtual abstractions of physical objects that can be repositioned, reorganized, and adapted to the task and device. However, designing such interfaces is currently manual and highly device-specific. We present Generative Proxy, a method for automatically generating proxy-based interfaces from three specifications: scene, intent, and device capabilities. We formulate generation as a constrained synthesis problem that first produces valid interfaces for the target device and task, then ranks candidates using semantic and articulatory distance inspired by direct manipulation theory. We demonstrate Generative Proxy across diverse scenes, device profiles, and user intents. Expert evaluation shows initial evidence that generated proxy UIs are useful and usable, highlighting proxy-based abstraction as a promising interaction paradigm for future AR glasses.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Generative Proxy
proxy-based interfaces
AR interaction
constrained synthesis
semantic distance
🔎 Similar Papers
No similar papers found.
