Bridging Handheld and Teleoperated Supervision for Contact-Rich Manipulation via State-Gated Experts

2026-06-25

AI Summary

This work addresses the limitations of handheld data collection—which captures only human observation actions and lacks dynamic feasibility during contact-sensitive phases—and the high cost of full-task teleoperation. To bridge this gap, the authors propose BRIDGE, a method that combines low-cost handheld data with a small number of staged teleoperated demonstrations. BRIDGE employs a state-gated mixture-of-experts architecture that dynamically routes the robot’s current state to the appropriate task-phase-specific expert policy, enabling phase-adaptive fusion of observed and desired actions. By integrating diffusion-based policy experts with a state-conditioned gating mechanism, BRIDGE achieves significant performance gains: across three contact-intensive manipulation tasks, it improves task success rates by up to 36.7% compared to a baseline using only handheld data.

Abstract

Handheld data collection systems, such as the Universal Manipulation Interface (UMI), enable scalable data collection across diverse environments but only capture observed actions rather than the desired actions executed by a robot controller. In contrast, teleoperation captures desired actions directly, but is prohibitively time-consuming to collect. We revisit this trade-off through the lens of action validity across task phases. We observe that handheld trajectories provide valid supervision in tolerant, free-space phases, but lack dynamic feasibility in contact-sensitive phases, where tracking observed trajectories at high stiffness produces large, unsafe contact forces. We study the interaction between these two supervision types for contact-rich manipulation and find that training policies that combine handheld data with a small number of targeted teleoperated demonstrations provide an efficient hybrid strategy. Specifically, rather than teleoperating the entire task, we only collect partial teleoperated demonstrations for task segments where base handheld policies fail. However, naively mixing handheld and teleoperated phase-specific data yields worse performance than training on handheld data alone. To address this mismatch between observed and desired supervision, we propose Bi-modal Routing for Imitation Data via Gated Experts (BRIDGE), a mixture of diffusion policy experts that routes between specialist task phase heads conditioned on the current robot state. Notably, our approach enables task-phase specific use of desired actions during contact sensitive segments and improves success rates over handheld-only baselines by up to 36.7% across three contact-rich manipulation tasks.

Problem

• contact-rich manipulation
• handheld data
• teleoperation
• action validity
• supervision mismatch

Innovation

• state-gated experts
• contact-rich manipulation
• hybrid imitation learning
• teleoperation
• diffusion policy