HUI360: A 360° Egocentric Dataset and Baselines for Human-Robot Interaction Anticipation

2026-08-11

AI Summary

This work addresses the challenge of enabling robots to proactively anticipate human intentions for natural human-robot interaction in real-world environments. To this end, the authors introduce HUI360, the first large-scale in-the-wild 360° egocentric human-robot interaction dataset, accompanied by an efficient annotation pipeline that combines automated labeling with manual correction. The dataset comprises over one million high-quality interaction annotations and is further expanded to six million labels in the SSUP-HRI benchmark. Integrating 360° panoramic video capture, 2D human pose estimation, facial landmark detection, and instance segmentation, the study establishes an end-to-end framework for interaction recognition and intention prediction. The authors publicly release the dataset, code, and baseline models, significantly advancing research and evaluation in human intention anticipation for human-robot interaction.

Abstract

As robots increasingly operate in human-populated environments, anticipating human intentions is essential for enabling proactive and socially aware behavior. Automatic anticipation of human-robot interactions is thus emerging as a crucial perception challenge for embodied agents. To this end, we introduce HUI360, the largest dataset for human-robot interaction anticipation in the wild and its set of baselines. The dataset was collected from a mobile robot, in the wild, over multiple days within a 3-month period, and in several environments, capturing natural, spontaneous behaviors from both passersby and users, and encompassing a diverse range of individuals. This variety enables evaluating and improving the generalization capabilities of interaction anticipation models. We designed a pipeline and share code for automatic interaction annotation in arbitrary 360-degree equirectangular videos, along with interfaces for manual refinement. Using this pipeline, we release the HUI360 open set of 1M pre-processed annotations, including detailed 2D poses, facial keypoints, and segmentation masks, obtained using state-of-the-art computer vision methods and manually curated to ensure high-quality tracking and interaction annotation. Additionally, we release the raw panoptic 360-degree images captured from the robot's egocentric viewpoint (on demand, for research purpose only in compliance with GDPR). Finally, we establish benchmark baselines for interaction anticipation, including the first cross-dataset evaluations for this task: to this end, we also release 6M annotations for another existing in-the-wild outdoor dataset collected from a mobile robot (SSUP-HRI). Dataset and code can be found at https://hucebot.github.io/hui360.

Problem

• human-robot interaction anticipation
• egocentric vision
• 360-degree video
• socially aware robotics
• interaction prediction

Innovation

• 360-degree egocentric vision
• human-robot interaction anticipation
• in-the-wild dataset
• automatic interaction annotation
• cross-dataset benchmark