HABIT: Human-Aware Behavior and Interaction Training Dataset for Robot Manipulation

2026-06-30

AI Summary

This work addresses the scarcity of human-in-the-loop scenarios in existing robotic manipulation datasets, which hinders the development of human-aware behavioral policies. To bridge this gap, the authors introduce the first large-scale robotic manipulation dataset that explicitly treats human presence as a dimension of diversity. Organized around three human-robot interaction roles—collaboration, coexistence, and supervision—the dataset comprises over 10,000 video demonstrations (160 hours) across 60 tasks, collected through large-scale, multimodal human demonstrations with detailed role annotations. Policies trained on this dataset exhibit human-perceptive capabilities such as spatiotemporal coordination, proactive avoidance, and gesture understanding, and demonstrate rapid generalization to novel human-robot interaction tasks, thereby advancing intelligent robot behavior in shared human environments.

Abstract

Large-scale demonstration datasets have been central to recent progress in general-purpose robot policies. However, existing datasets are collected in human-absent settings, and policies trained on such data may perform tasks competently in isolation but fail to exhibit human-aware behaviors. To address this gap, we introduce HABIT, a large-scale robot demonstration dataset for human-present environments. We organize tasks into three roles capturing distinct modes of human-robot interaction: Collaborator, where human and robot jointly accomplish a task; Coworker, where they pursue separate tasks in a shared space; and Supervisor, where the human directs the robot. The dataset comprises over 10K episodes and over 160 hours across 60 tasks. Our experiments show that training on human-present data elicits human-aware behaviors that robot-only data fails to produce: spatiotemporal synchronization in Collaborator tasks, yielding in Coworker tasks, and gesture grounding in Supervisor tasks. Moreover, training on HABIT enables rapid adaptation to new human-robot interaction tasks. By introducing human presence as a new axis of dataset diversity, HABIT extends robot policies to environments shared with humans.

Problem

• human-aware behavior
• robot manipulation
• human-robot interaction
• demonstration dataset

Innovation

• human-aware robotics
• robot demonstration dataset
• human-robot interaction
• behavioral adaptation
• spatiotemporal synchronization