GHOST in the Robots: Real-Time Exocentric Dual-Robot VR Teleoperation from Onboard Cameras

2026-08-29

AI Summary

GHOST系统通过实时整合机器人摄像头数据和基于学习的场景补全，解决了单人同时操控两台移动操作机器人的问题，提高了操作成功率和效率。

Abstract

Teleoperating multiple robots simultaneously enables additional views and coordinated control. Yet, it poses fundamental challenges: the system must present sensor data cohesively and allow operators to manage multiple robot bases, arms, and cameras while maintaining low latency. Current multi-robot teleoperation systems require multiple operators, rely on autonomy, or restrict operators to high-level commands. We present GHOST: an open-source VR teleoperation system that enables single operator control of two mobile manipulators via direct lowlevel commands using only onboard sensing. GHOST creates an exocentric 3D workspace by aligning real-time point clouds from the robots' RGB-D cameras, where scene coverage is improved through learning-based completion to aid operator spatial awareness. For control, the operator uses a mode-switching architecture to command either robot individually or both robots simultaneously. Experiments with 15 novice participants demonstrate 1.6-4x the success rate of an off-the-shelf tablet interface. For experts across nine challenging dual-robot tasks, our system enabled completion of two tasks that were infeasible with the tablet, and was 1.47x faster on average than the tablet. Website and code: https://h2r.github.io/GHOST/.

Problem

• multi-robot teleoperation
• low latency
• sensor data integration

Innovation

• exocentric 3D workspace
• onboard sensing
• real-time point clouds
• learning-based completion
• mode-switching architecture