At a Glance to Your Fingertips: Enabling Direct Manipulation of Distant Objects Through SightWarp

2025-08-06

AI Summary

To address the challenge of direct hand-based interaction with distant objects in 3D user interfaces, this paper introduces SightWarp—a vision-guided near-field proxy mechanism leveraging eye-hand coordination. Upon fixating on a distant target, users trigger real-time generation of an interactive, scaled proxy by moving their hand into the field of view or redirecting gaze toward the hand; this proxy supports natural, direct 6-DOF gestural manipulation. Our key contribution is the first integration of visual guidance with dynamic proxy instantiation—eliminating indirect gaze-and-pinch interactions while preserving proprioceptive feedback and operational efficiency. SightWarp fuses gaze tracking, hand pose estimation, and spatial mapping to enable overview-detail navigation and miniature-world interaction. A user study demonstrates that SightWarp significantly outperforms baseline methods in 3D docking tasks (p < 0.01), reducing task completion time by 27.4% and substantially improving subjective usability and immersion.

Technology Category

• Computer Vision: 3D Computer Vision
• Humans and AI: Interaction Techniques and Devices
• Intelligent Robots: Manipulation

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

In 3D user interfaces, reaching out to grab and manipulate something works great until it is out of reach. Indirect techniques like gaze and pinch offer an alternative for distant interaction, but do not provide the same immediacy or proprioceptive feedback as direct gestures. To support direct gestures for faraway objects, we introduce SightWarp, an interaction technique that exploits eye-hand coordination to seamlessly summon object proxies to the user's fingertips. The idea is that after looking at a distant object, users either shift their gaze to the hand or move their hand into view-triggering the creation of a scaled near-space proxy of the object and its surrounding context. The proxy remains active until the eye-hand pattern is released. The key benefit is that users always have an option to immediately operate on the distant object through a natural, direct hand gesture. Through a user study of a 3D object docking task, we show that users can easily employ SightWarp, and that subsequent direct manipulation improves performance over gaze and pinch. Application examples illustrate its utility for 6DOF manipulation, overview-and-detail navigation, and world-in-miniature interaction. Our work contributes to expressive and flexible object interactions across near and far spaces.

Problem

• Enabling direct manipulation of distant 3D objects
• Improving interaction immediacy and proprioceptive feedback
• Bridging near and far space interactions seamlessly

Innovation

• SightWarp enables direct manipulation of distant objects
• Uses eye-hand coordination to create near-space proxies
• Improves performance over gaze and pinch techniques