Refine Any Object in Any Scene

2025-06-30

AI Summary

In scene reconstruction, camera trajectories often result in occluded or missing views of individual objects, hindering simultaneous high-fidelity object modeling and global scene consistency. To address this, we propose RAISE, the first framework to integrate 3D generative priors into object-level inpainting. RAISE employs a two-stage refinement pipeline for joint geometric and appearance recovery: (1) a generative model serves as a proxy for the degraded object, enabling robust 7-DOF pose alignment; (2) enhanced spatial registration—incorporating both geometric correspondence and pose-aware regularization—ensures geometric completeness and photorealistic texture synthesis under unseen viewpoints. RAISE enables localized enhancement of arbitrary objects within complex scenes. Extensive experiments demonstrate state-of-the-art performance on novel-view synthesis and geometric completion tasks. The code is publicly available.

Technology Category

• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Viewpoint missing of objects is common in scene reconstruction, as camera paths typically prioritize capturing the overall scene structure rather than individual objects. This makes it highly challenging to achieve high-fidelity object-level modeling while maintaining accurate scene-level representation. Addressing this issue is critical for advancing downstream tasks requiring detailed object understanding and appearance modeling. In this paper, we introduce Refine Any object In any ScenE (RAISE), a novel 3D enhancement framework that leverages 3D generative priors to recover fine-grained object geometry and appearance under missing views. Starting from substituting degraded objects with proxies, via a 3D generative model with strong 3D understanding, RAISE progressively refines geometry and texture by aligning each proxy to its degraded counterpart in 7-DOF pose, followed by correcting spatial and appearance inconsistencies via registration-constrained enhancement. This two-stage refinement ensures the high-fidelity geometry and appearance of the original object in unseen views while maintaining consistency in spatial positioning, observed geometry, and appearance. Extensive experiments on challenging benchmarks show that RAISE significantly outperforms state-of-the-art methods in both novel view synthesis and geometry completion tasks. RAISE is made publicly available at https://github.com/PolySummit/RAISE.

Problem

• Recover fine-grained object geometry under missing views
• Enhance object appearance modeling in 3D scenes
• Maintain scene consistency while refining individual objects

Innovation

• Leverages 3D generative priors for enhancement
• Two-stage refinement for geometry and texture
• Aligns proxies via 7-DOF pose correction