COBALT: Crowdsourcing Robot Learning via Cloud-Based Teleoperation with Smartphones

2026-05-18

AI Summary

This work addresses the scarcity of high-quality demonstration data in robot imitation learning by introducing a cloud-based crowdsourced teleoperation platform that enables large-scale, low-cost, and highly synchronized remote control using ubiquitous devices such as smartphones. The platform integrates structured training protocols with real-time quality monitoring and employs a vectorized environment, efficient video streaming, in-memory caching, and load-balancing architecture to support multimodal inputs—including smartphones, VR headsets, and 3D mice. Over five days across nine countries, the system collected more than 7,500 high-fidelity demonstrations (exceeding 50 hours of data). It scales efficiently from supporting eight concurrent users on a single GPU to 256 clients across eight GPUs, and the gathered data significantly enhances the training performance of state-of-the-art imitation learning algorithms.

Abstract

The scarcity of large-scale, high-quality demonstration data remains a bottleneck in scaling imitation learning for robotic manipulation. We present COBALT, a teleoperation platform designed to democratize robot learning at scale both in simulation and in the real world. By leveraging vectorized environments, our scalable, load-balanced infrastructure supports concurrent teleoperation by multiple users on a single GPU, yielding a significant reduction in teleoperation cost. Operators can connect from nearly anywhere on Earth using commonly available devices, including single or dual smartphones, VR headsets, 3D mice, and keyboards. An inmemory data cache and efficient video streaming keep control and rendering synchronous, sustaining dozens of concurrent users at 20 Hz with sub-100 ms end-to-end latency for up to 8 concurrent users per GPU. We also demonstrate stable operation supporting 256 simulated clients across 8 GPUs, underscoring the system's ability to scale across hardware and within individual servers. We perform a comprehensive user study showing that phone-based teleoperation performs comparably to or better than specialized hardware, enabling faster, more ergonomic data collection. To ensure data quality, COBALT logs a suite of real-time metrics to automatically filter suboptimal demonstrations. We further demonstrate that a structured user training curriculum significantly improves data collection quality. Guided by insights from our user study, we crowdsource the collection of a large-scale, high-quality pilot dataset with 7500+ demonstrations (50+ hours) collected with smartphones across nine countries over five days. We validate the dataset's quality by training state-of-the-art imitation learning algorithms. Please visit \href{https://cobalt-teleop.github.io/}{cobalt-teleop.github.io} for more details.

Problem

• imitation learning
• robotic manipulation
• demonstration data
• teleoperation
• data scarcity

Innovation

• cloud-based teleoperation
• crowdsourced robot learning
• vectorized environments
• smartphone-based control
• imitation learning