Scene and Human in One World: Reconstruction in a Feedforward Pass

2026-06-26

AI Summary

This work addresses the challenges of scale ambiguity, misalignment between humans and scenes, and occlusion interference in human reconstruction from dynamic scenes captured by a moving monocular camera. To tackle these issues, the authors propose SHOW—a framework that jointly infers human meshes and scene geometry in a unified metric space through a feed-forward pipeline. The method integrates the semantic structure and scale priors of parametric human models and employs a promptable mask mechanism to flexibly specify target individuals. Mutual guidance between human and scene representations enhances spatial alignment and metric scale consistency. Experiments demonstrate that SHOW significantly improves metric-scale reconstruction accuracy, human-scene alignment quality, and overall robustness in complex scenarios involving multiple people, severe occlusions, and cluttered backgrounds.

Abstract

Reconstructing humans in dynamic scenes from moving monocular cameras remains challenging due to scale ambiguity, human-scene misalignment, and occlusion interference. Rather than treating human mesh recovery and scene reconstruction as separate tasks, we believe that accurate human-scene reconstruction requires the two tasks to mutually inform each other: parametric human models offer semantic structure and metric-scale priors, while scene geometry provides spatial context for human localization and alignment. Built on this insight, we introduce SHOW, a mask-promptable human mesh recovery framework that couples feed-forward 3D scene reconstruction with Human Mesh Recovery in a unified metric space. SHOW injects human semantics and scale priors from parametric human models into normalized point-map prediction, enabling metric-scale scene reconstruction from inherently scale-ambiguous monocular input. In turn, the recovered scene geometry constrains human mesh estimation, encouraging spatially consistent human placement and improved human-scene alignment. To handle complex multi-person and cluttered scenes, SHOW further incorporates a promptable masking mechanism that enables flexible target-human selection while suppressing background distractions and occlusion interference. Through joint training, the model learns both human-aware geometric features and geometry-constrained human features, producing aligned metric-scale reconstructions from monocular human-centric videos. Extensive experiments demonstrate that SHOW improves metric-scale consistency, human-scene alignment, and reconstruction accuracy under challenging camera motion, occlusion, and cluttered backgrounds.

Problem

• scale ambiguity
• human-scene misalignment
• occlusion interference
• monocular reconstruction
• dynamic scenes

Innovation

• human-scene reconstruction
• metric-scale recovery
• feedforward framework
• promptable masking
• monocular video