NeRF-MIR: Towards High-Quality Restoration of Masked Images with Neural Radiance Fields

2026-01-24

AI Summary

This work addresses the significant degradation in NeRF reconstruction quality caused by occluded or corrupted input images by proposing NeRF-MIR, the first method to apply NeRF to 3D scene inpainting from occluded views. NeRF-MIR introduces three key innovations: a Patch-based Entropy-guided Ray Emission (PERE) strategy, a Progressive Iterative Reconstruction (PIRE) mechanism, and a dynamic weighted loss function. Together, these components enable effective self-supervised restoration of occluded regions while leveraging multi-view consistency to enhance reconstruction fidelity. The authors also contribute the first benchmark dataset for this task, comprising three occlusion-specific subsets. Extensive experiments demonstrate that NeRF-MIR consistently outperforms existing approaches on both synthetic and real-world occluded data, validating its effectiveness and superiority in occlusion-aware 3D scene reconstruction.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Neural Radiance Fields (NeRF) have demonstrated remarkable performance in novel view synthesis. However, there is much improvement room on restoring 3D scenes based on NeRF from corrupted images, which are common in natural scene captures and can significantly impact the effectiveness of NeRF. This paper introduces NeRF-MIR, a novel neural rendering approach specifically proposed for the restoration of masked images, demonstrating the potential of NeRF in this domain. Recognizing that randomly emitting rays to pixels in NeRF may not effectively learn intricate image textures, we propose a \textbf{P}atch-based \textbf{E}ntropy for \textbf{R}ay \textbf{E}mitting (\textbf{PERE}) strategy to distribute emitted rays properly. This enables NeRF-MIR to fuse comprehensive information from images of different views. Additionally, we introduce a \textbf{P}rogressively \textbf{I}terative \textbf{RE}storation (\textbf{PIRE}) mechanism to restore the masked regions in a self-training process. Furthermore, we design a dynamically-weighted loss function that automatically recalibrates the loss weights for masked regions. As existing datasets do not support NeRF-based masked image restoration, we construct three masked datasets to simulate corrupted scenarios. Extensive experiments on real data and constructed datasets demonstrate the superiority of NeRF-MIR over its counterparts in masked image restoration.

Problem

• Neural Radiance Fields
• Masked Image Restoration
• 3D Scene Reconstruction
• Corrupted Images
• Novel View Synthesis

Innovation

• Neural Radiance Fields
• masked image restoration
• patch-based entropy
• progressive iterative restoration
• dynamic loss weighting