UniPR: Unified Object-level Real-to-Sim Perception and Reconstruction from a Single Stereo Pair

2026-03-19

AI Summary

This work proposes UniPR, an end-to-end framework that simultaneously performs cross-category object perception and reconstruction from a single stereo image pair, overcoming the inefficiency and error accumulation inherent in conventional multi-module pipelines. To address scale ambiguity, geometric constraints are explicitly incorporated, while a pose-aware shape representation eliminates the need for category-specific priors. The authors further introduce LVS6D, a large-scale stereo dataset designed to support training and evaluation of such joint tasks. UniPR reconstructs all objects in a scene within a single forward pass, achieving significantly faster inference while accurately preserving real-world physical scales.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Perceiving and reconstructing objects from images are critical for real-to-sim transfer tasks, which are widely used in the robotics community. Existing methods rely on multiple submodules such as detection, segmentation, shape reconstruction, and pose estimation to complete the pipeline. However, such modular pipelines suffer from inefficiency and cumulative error, as each stage operates on only partial or locally refined information while discarding global context. To address these limitations, we propose UniPR, the first end-to-end object-level real-to-sim perception and reconstruction framework. Operating directly on a single stereo image pair, UniPR leverages geometric constraints to resolve the scale ambiguity. We introduce Pose-Aware Shape Representation to eliminate the need for per-category canonical definitions and to bridge the gap between reconstruction and pose estimation tasks. Furthermore, we construct a large-vocabulary stereo dataset, LVS6D, comprising over 6,300 objects, to facilitate large-scale research in this area. Extensive experiments demonstrate that UniPR reconstructs all objects in a scene in parallel within a single forward pass, achieving significant efficiency gains and preserves true physical proportions across diverse object types, highlighting its potential for practical robotic applications.

Problem

• real-to-sim transfer
• object-level perception
• 3D reconstruction
• modular pipeline
• cumulative error

Innovation

• end-to-end
• object-level reconstruction
• pose-aware shape representation
• real-to-sim transfer
• stereo perception