MutualNeRF: Improve the Performance of NeRF under Limited Samples with Mutual Information Theory

2025-05-16

AI Summary

This work addresses the degradation of Neural Radiance Field (NeRF) reconstruction quality under sparse-view settings. To overcome limitations of existing methods—such as the absence of a unified prior model and reliance on ground-truth supervision—we propose a novel few-shot training framework grounded in mutual information (MI) theory. Specifically, we introduce MI as a differentiable, unsupervised, and unified metric that jointly captures image-level semantics and pixel-level correlations. Building upon this, we design a greedy view-selection strategy for optimal viewpoint sampling and incorporate a plug-and-play MI-maximization regularization module. Extensive experiments demonstrate that our approach consistently surpasses state-of-the-art methods across diverse few-shot configurations, achieving significant improvements in PSNR and SSIM. Moreover, it exhibits superior view-selection efficiency and strong generalization capability.

Technology Category

• Machine Learning: Multi-instance/Multi-view Learning
• Computer Vision: Multi-modal Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

This paper introduces MutualNeRF, a framework enhancing Neural Radiance Field (NeRF) performance under limited samples using Mutual Information Theory. While NeRF excels in 3D scene synthesis, challenges arise with limited data and existing methods that aim to introduce prior knowledge lack theoretical support in a unified framework. We introduce a simple but theoretically robust concept, Mutual Information, as a metric to uniformly measure the correlation between images, considering both macro (semantic) and micro (pixel) levels. For sparse view sampling, we strategically select additional viewpoints containing more non-overlapping scene information by minimizing mutual information without knowing ground truth images beforehand. Our framework employs a greedy algorithm, offering a near-optimal solution. For few-shot view synthesis, we maximize the mutual information between inferred images and ground truth, expecting inferred images to gain more relevant information from known images. This is achieved by incorporating efficient, plug-and-play regularization terms. Experiments under limited samples show consistent improvement over state-of-the-art baselines in different settings, affirming the efficacy of our framework.

Problem

• Enhance NeRF performance with limited samples using Mutual Information Theory
• Select optimal viewpoints by minimizing mutual information for sparse views
• Improve few-shot synthesis by maximizing mutual information with ground truth

Innovation

• Uses Mutual Information Theory for NeRF enhancement
• Selects viewpoints by minimizing mutual information
• Maximizes mutual information with plug-and-play regularization