HATS: A Human-Agent Teleoperation System for Multi-Arm Data Collection

2026-06-15

AI Summary

This study addresses the challenges of high cognitive load on single operators and substantial coordination overhead in multi-operator setups for multi-arm teleoperation tasks. The authors propose a human–robot collaborative teleoperation framework wherein a human directly controls two master arms, while two auxiliary arms are autonomously managed by a training-free multimodal large language model (MLLM) agent to execute subtasks, with real-time intervention enabled via voice commands. This approach pioneers the integration of MLLM agents into multi-arm teleoperation for data collection, achieving decoupled control spaces and natural human–robot interaction. The system maintains high operational efficiency while significantly enhancing scalability. Experimental results demonstrate that the proposed method achieves data collection success rates and efficiency comparable to those of expert two-human teams, and the collected data effectively supports downstream training of multi-arm collaborative policies.

Abstract

Many real-world manipulation scenarios, such as handling complex collaborative tasks and dealing with large workspaces, require coordination of more than two robotic arms. Consequently, an effective multi-arm teleoperation system is required to collect demonstrations for training coordinated multi-arm manipulation policies. However, existing teleoperation frameworks mainly focus on single-operator or multi-operator setups, facing a practical trade-off between the cognitive load placed on a single operator and the coordination cost incurred by multiple operators. To address this problem, we introduce HATS, a human-agent teleoperation system that enables a single human operator, assisted by an MLLM-based agent, to collect data for multi-arm manipulation tasks. Our system decouples the control space: two primary arms are directly teleoperated by the human, while two assistive arms are controlled by a training-free agent that handles sub-tasks. In addition, the human operator can use voice commands to prevent collisions and correct assistive arm behaviors during execution. Extensive evaluations demonstrate that HATS achieves data collection efficiency and success rates comparable to expert dual-human teams. Moreover, downstream policy evaluations demonstrate the efficacy and quality of the data collected through HATS.

Problem

• multi-arm teleoperation
• data collection
• human-robot collaboration
• cognitive load
• coordination cost

Innovation

• multi-arm teleoperation
• human-agent collaboration
• MLLM-based agent
• decoupled control
• voice-guided correction