OmniRobotHome: A Multi-Camera Platform for Real-Time Multiadic Human-Robot Interaction

2026-04-30

AI Summary

This work addresses the challenge that existing systems struggle to support concurrent human–robot collaboration in real-world domestic environments involving multiple people and robots operating under tight spatiotemporal coupling, primarily due to the lack of real-time, occlusion-robust, room-scale 3D perception. To overcome this limitation, the authors present a room-scale experimental platform integrating 48 hardware-synchronized RGB cameras to enable markerless, occlusion-resistant real-time 3D tracking of multiple humans and objects. Within a unified world coordinate system, the system coordinates two Franka robotic arms for collaborative tasks. This framework represents the first integration of room-scale perception and actuation, enabling long-term modeling of human–robot interaction. Evaluations in shared-environment safety and anticipatory robotic assistance demonstrate that the proposed real-time perception and behavioral memory mechanisms significantly enhance overall system performance.

Abstract

Human-robot collaboration has been studied primarily in dyadic or sequential settings. However, real homes require multiadic collaboration, where multiple humans and robots share a workspace, acting concurrently on interleaved subtasks with tight spatial and temporal coupling. This regime remains underexplored because close-proximity interaction between humans, robots, and objects creates persistent occlusion and rapid state changes, making reliable real-time 3D tracking the central bottleneck. No existing platform provides the real-time, occlusion-robust, room-scale perception needed to make this regime experimentally tractable. We present OmniRobotHome, the first room-scale residential platform that unifies wide-area real-time 3D human and object perception with coordinated multi-robot actuation in a shared world frame. The system instruments a natural home environment with 48 hardware-synchronized RGB cameras for markerless, occlusion-robust tracking of multiple humans and objects, temporally aligned with two Franka arms that act on live scene state. Continuous capture within this consistent frame further supports long-horizon human behavior modeling from accumulated trajectories. The platform makes the multiadic collaboration regime experimentally tractable. We focus on two central problems: safety in shared human-robot environments and human-anticipatory robotic assistance, and show that real-time perception and accumulated behavior memory each yield measurable gains in both.

Problem

• multiadic human-robot interaction
• real-time 3D tracking
• occlusion-robust perception
• shared workspace collaboration
• room-scale perception

Innovation

• multiadic human-robot interaction
• real-time 3D tracking
• occlusion-robust perception
• room-scale multi-camera system
• coordinated multi-robot actuation