Decoupled Hands: An Approach for Aligning Perspectives in Collaborative Mixed Reality

2025-03-15

AI Summary

In mixed reality (MR) co-located collaboration, differing user viewpoints cause occlusion and spatial reference ambiguity (e.g., inconsistent interpretation of “left”), hindering effective communication. To address this without virtual repositioning, we propose a dynamic viewpoint alignment method grounded in hand-based semantic anchoring. Our core innovation is the “decoupled hands” mechanism: leveraging real-time hand pose estimation, collaborative coordinate system mapping, and dynamic scene graph rebinding, it adaptively transforms local virtual objects and remote users’ hand representations—while preserving physical positions—to achieve semantic-level spatial alignment anchored to hand poses. Evaluation via a prototype system demonstrates significant reductions in instruction misinterpretation, improved task efficiency, and enhanced subjective naturalness. This work establishes a new paradigm for MR co-located collaboration that simultaneously ensures precision, natural interaction, and zero physical intrusion.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Multiagent Systems: Coordination and Collaboration

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

When collaborating relative to a shared 3D virtual object in mixed reality (MR), users may experience communication issues arising from differences in perspective. These issues include occlusion (e.g., one user not being able to see what the other is referring to) and inefficient spatial references (e.g.,"to the left of this"may be confusing when users are positioned opposite to each other). This paper presents a novel technique for automatic perspective alignment in collaborative MR involving co-located interaction centered around a shared virtual object. To align one user's perspective on the object with a collaborator's, a local copy of the object and any other virtual elements that reference it (e.g., the collaborator's hands) are dynamically transformed. The technique does not require virtual travel and preserves face-to-face interaction. We created a prototype application to demonstrate our technique and present an evaluation methodology for related MR collaboration and perspective alignment scenarios.

Problem

• Addresses communication issues in collaborative mixed reality
• Resolves occlusion and spatial reference inefficiencies
• Develops automatic perspective alignment technique

Innovation

• Dynamic transformation of local object copies
• Automatic perspective alignment in MR
• Preserves face-to-face interaction