DeSeG: Decoupling Semantic Intent and Geometric Constraints for Physically Plausible Human-Scene Interaction

2026-07-06

AI Summary

Existing generative models struggle to simultaneously satisfy semantic intent and geometric constraints when synthesizing human-scene interactions, often producing motions that either violate textual instructions or exhibit physically implausible interpenetrations. To address this, this work proposes DeSeG, a hierarchical framework that explicitly decouples semantic planning from geometric execution. DeSeG first employs a residual semantic planner to interpret text instructions and generate target poses, then leverages a diffusion-based executor grounded in differentiable repulsive potential fields to synthesize collision-free motions. This design mitigates the model’s reliance on spatial shortcut biases, yielding state-of-the-art performance on the Lingo dataset—reducing average scene penetration by 47% and improving semantic alignment by 29%.

Abstract

Synthesizing physically plausible human-scene interactions (HSI) remains a critical challenge in computer vision and the development of human avatars. Although recent generative models enable diverse motion synthesis, they suffer from an inductive bias referred to as semantic-geometric entanglement. Because spatial constraints often strongly correlate with specific actions in training data, monolithic models will learn the shortcut bias, aggressively overriding the semantic intent when faced with strict geometric cues. Furthermore, this entanglement exacerbates physical hallucinations, such as body-scene penetrations. To address these limitations, we propose DeSeG, a hierarchical framework that explicitly decouples semantic intent from geometric constraints. First, we introduce a Residual Semantic Planner that encodes textual instructions and canonicalized goal voxels into a compact latent space, enabling fine-grained semantic control independent of spatial trajectories. Second, we propose a physics regularized diffusion executor that incorporates differentiable repulsive potential fields directly into the diffusion objective, enforcing collision-aware motion generation. Extensive experiments on the Lingo dataset demonstrate that DeSeG achieves state-of-the-art performance, reducing mean scene penetration by 47% and improving semantic alignment by 29% over the SOTA baselines.

Problem

• human-scene interaction
• semantic-geometric entanglement
• physical plausibility
• collision avoidance
• generative models

Innovation

• semantic-geometric decoupling
• human-scene interaction
• diffusion model
• physics-aware generation
• residual semantic planner