HSImul3R: Physics-in-the-Loop Reconstruction of Simulation-Ready Human-Scene Interactions

2026-03-16

AI Summary

This work addresses the limitations of existing methods for human-scene interaction reconstruction, which often produce visually plausible but physically implausible results that are unstable in simulation and unsuitable for embodied intelligence. The authors propose a unified framework that jointly reconstructs dynamic human motion and scene geometry from sparse-view images or monocular video, introducing for the first time a physics-based closed-loop feedback mechanism. In this loop, human motion is optimized forward via scene-aware reinforcement learning, while scene geometry is refined backward using simulation-derived rewards to enforce gravitational stability and interaction feasibility. The method yields the first simulation-ready, interaction-aware reconstructions directly deployable on real humanoid robots and introduces HSIBench, a new benchmark featuring diverse objects and interactions.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Human-Robot Interaction
• Computer Vision: Low Level & Physics-based Vision

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

We present HSImul3R, a unified framework for simulation-ready 3D reconstruction of human-scene interactions (HSI) from casual captures, including sparse-view images and monocular videos. Existing methods suffer from a perception-simulation gap: visually plausible reconstructions often violate physical constraints, leading to instability in physics engines and failure in embodied AI applications. To bridge this gap, we introduce a physically-grounded bi-directional optimization pipeline that treats the physics simulator as an active supervisor to jointly refine human dynamics and scene geometry. In the forward direction, we employ Scene-targeted Reinforcement Learning to optimize human motion under dual supervision of motion fidelity and contact stability. In the reverse direction, we propose Direct Simulation Reward Optimization, which leverages simulation feedback on gravitational stability and interaction success to refine scene geometry. We further present HSIBench, a new benchmark with diverse objects and interaction scenarios. Extensive experiments demonstrate that HSImul3R produces the first stable, simulation-ready HSI reconstructions and can be directly deployed to real-world humanoid robots.

Problem

• human-scene interaction
• physics simulation
• 3D reconstruction
• embodied AI
• perception-simulation gap

Innovation

• physics-in-the-loop
• simulation-ready reconstruction
• human-scene interaction
• bi-directional optimization
• reinforcement learning