Can Single-View Mesh Reconstruction Generalize to Robot Camera Rotation?

2026-06-22

AI Summary

Existing single-view mesh reconstruction methods exhibit poor generalization under camera rotation due to their reliance on viewpoint priors, often resulting in 3D inconsistencies, erroneous scene layouts, and violations of physical constraints. This work proposes the first evaluation protocol specifically designed for single-view reconstruction under camera rotation, enabling systematic assessment of depth estimation, object meshes, scene layout, and physical plausibility. A two-stage pipeline built upon SAM3D and FoundationPose—augmented with ICP registration, monocular depth estimation, and gravity alignment—significantly enhances robustness. Furthermore, a novel gravity-aware refinement strategy reduces layout orientation error by 47.1% compared to single-stage approaches.

Abstract

Single-view mesh reconstruction predicts object meshes and spatial layouts from a single observation, making it attractive for fast robot spatial reasoning and real-to-sim digital twins. However, robot-mounted cameras naturally rotate during manipulation and navigation, while learned single-view reconstruction models often rely on view-dependent priors and may generalize poorly to out-of-distribution camera rotations. Such rotations can introduce 3D inconsistencies, incorrect layouts, and violations of physical constraints, but this failure mode remains under-evaluated. We introduce an evaluation protocol with controlled axis-wise roll, pitch, and yaw sweeps to trace errors in monocular depth estimation (MDE), canonical object meshes, camera-space layout, and physical plausibility within a representative SAM3D-style pipeline. On the Aria Digital Twin dataset and a real Franka wrist-camera sequence, camera rotations induce MDE distortion, layout drift, and collision penetration, while canonical mesh predictions remain relatively stable. A two-stage SAM3D+FoundationPose pipeline is more robust than one-stage feed-forward layout prediction, and our Gravity-Aware Refinement reduces one-stage pairwise ICP-based layout-orientation error by 47.1$\%$. Our evaluation reveals that current single-view mesh reconstruction methods generalize poorly to robot camera rotation, and suggests that explicit gravity cues are important for reliable robotic single-view mesh reconstruction.

Problem

• single-view mesh reconstruction
• camera rotation
• robot perception
• 3D reconstruction generalization
• physical plausibility

Innovation

• single-view mesh reconstruction
• camera rotation robustness
• gravity-aware refinement
• robotic spatial reasoning
• 3D layout consistency