SceneBot: Contact-Prompted General Humanoid Whole Body Tracking with Scene-Interaction

2026-06-25

AI Summary

This work addresses the challenge that existing humanoid reinforcement learning policies, while effective in free-space locomotion, struggle with physical ambiguities in contact-intensive tasks such as object interaction or walking on uneven terrain. To overcome this limitation, the authors propose SceneBot, a unified motion control framework that jointly conditions a single policy on action references and limb contact labels, enabling coherent control across free locomotion, terrain traversal, and whole-body manipulation. The key innovations include treating contact conditions as a universal interface for humanoid control and introducing a hindsight scene reconstruction method to automatically generate contact interaction data. Remarkably, training solely on 7.5 hours of reconstructed data enables the policy to generalize to unseen actions and environments, successfully executing complex long-horizon tasks such as carrying a box up stairs.

Abstract

Current humanoid reinforcement-learning policies excel at free-space motions but struggle with contact-rich tasks, as pure kinematic tracking cannot resolve the physical ambiguities of interacting with objects and uneven terrain. To address this, we introduce SceneBot, a unified motion-tracking framework capable of handling freespace locomotion, terrain traversal, and whole-body manipulation. SceneBot conditions a single policy on both reference motions and per-link contact labels, explicitly defining expected environmental interactions. To overcome the lack of annotated interaction data, we propose a hindsight scene reconstruction approach that infers scene-interaction graphs from retargeted human motion. Trained on 7.5 hours of this reconstructed, contact-rich data, SceneBot successfully generalizes to unseen motions and environments. Our results demonstrate that SceneBot is the first general framework to seamlessly unify free-space and contact-rich behaviors executing complex, long-horizon tasks like carrying a box upstairs and establishing contact conditioning as a powerful interface for humanoid control. All code and data will be open-sourced. More demos and information are available at: https://ericcsr.github.io/scenebot/

Problem

• humanoid control
• contact-rich tasks
• whole-body tracking
• scene interaction
• motion ambiguity

Innovation

• contact conditioning
• scene interaction
• whole-body tracking
• hindsight reconstruction
• humanoid control