Policy-as-Data: Learning Generalizable HOI Diffusion Models from Simulated Physics

2026-06-21

AI Summary

Existing human-object interaction (HOI) generation methods based on motion capture data exhibit limited generalization to unseen objects and struggle to maintain physical consistency over extended interactions. This work proposes a “policy-as-data” framework that leverages reinforcement learning within a physics simulator to generate task-oriented synthetic trajectories. To bridge the representational gap between simulated characters and standard parametric human body models, the authors design a coarse-to-fine retargeting pipeline, enabling the training of a generalizable HOI diffusion model. The resulting approach significantly improves generalization to novel objects and supports the synthesis of long-duration, physically plausible interactions with high dynamic diversity.

Abstract

Synthesizing realistic Human-Object Interactions (HOI) is critical for creating embodied avatars and functional virtual environments. However, current data-driven approaches primarily rely on motion capture datasets, which are expensive to scale and limited in functional diversity. Models trained with these datasets fail to generalize to unseen objects and maintain physical consistency over long horizons. In this paper, we propose a novel framework that leverages a physics simulator to overcome the data-scarcity bottleneck in HOI generation. Specifically, we propose a scalable pipeline, called \ours, which leverages policies trained with reinforcement learning in a physics simulator for task-oriented data generation and trains a generative model on the augmented dataset for generalizable HOI generation. To seamlessly utilize the synthetic data, we introduce a coarse-to-fine retargeting process that bridges the representation gap between the simplified model used in physics simulator and the standard parametric body models required for generative training. Validated through comprehensive experiments, our method demonstrates enhanced generalization to unseen objects and the capability of long-horizon generation, while exhibiting greater dynamic diversity and physical plausibility.

Problem

• Human-Object Interaction
• data scarcity
• generalization
• physical consistency
• motion capture

Innovation

• Policy-as-Data
• Human-Object Interaction
• Physics Simulation
• Reinforcement Learning
• Diffusion Models