Bimanual XR Specification of Relative and Absolute Assembly Hierarchies for Teleoperation

2026-03-02

AI Summary

This work addresses the lack of efficient and intuitive methods for expressing hierarchical structures involving both relative and absolute constraints in remote assembly tasks. The authors propose a bimanual extended reality (XR) interaction technique that enables users to construct nested constraint groups by grasping objects with each hand, assigning either relative constraints—where robot-optimized poses are computed—or absolute constraints—where user-specified poses are fixed—to each group. This approach introduces, for the first time in bimanual XR, hierarchical modeling of mixed 6DoF constraints, augmented with visual convex hull representations to generate robot-interpretable assembly specifications. By minimizing the need for manual specification of precise object poses, the method significantly enhances the efficiency and flexibility of remote teleoperation in complex assembly scenarios.

Technology Category

• Intelligent Robots: Manipulation
• Humans and AI: Interaction Techniques and Devices
• Constraint Satisfaction and Optimization: Mixed Discrete/Continuous Optimization

Application Category

• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

We present a bimanual XR interaction approach for specifying remote assembly tasks as hierarchies of relative and absolute object constraints that specify high-level teleoperation goals for robots. Grabbing one object in each hand creates a constraint group (visualized as a hull) and groups can be nested into hierarchies. Each group can be relative (with a robot-specifiable 6DoF pose) or absolute (with an author-specified fixed 6DoF pose) in relation to its parent. A relative group specifies a subassembly that can be constructed at a location chosen by the robot software for efficiency rather than mandated by the user.

Problem

• bimanual XR
• assembly hierarchies
• teleoperation
• object constraints
• 6DoF pose

Innovation

• bimanual XR
• assembly hierarchy
• relative constraint
• absolute constraint
• teleoperation