R-Meshfusion: Reinforcement Learning Powered Sparse-View Mesh Reconstruction with Diffusion Priors

2025-04-16

AI Summary

This work addresses geometric distortion and appearance inconsistency in multi-view mesh reconstruction under sparse-view settings. We propose a robust reconstruction framework integrating diffusion priors with reinforcement learning. Our method introduces: (1) a consensus diffusion module that leverages IQR filtering and variance-aware image fusion to convert diffusion model outputs into high-confidence pseudo-supervisory signals; and (2) a UCB-based online view selection strategy that dynamically optimizes observation sequences to enhance joint NeRF optimization efficiency. To the best of our knowledge, this is the first approach to enable trustworthy guidance of sparse-view NeRF training via diffusion priors. Experiments demonstrate significant improvements over state-of-the-art methods: a 32% reduction in Chamfer distance for geometric accuracy and a 4.1 dB PSNR gain in novel-view synthesis quality.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Mesh reconstruction from multi-view images is a fundamental problem in computer vision, but its performance degrades significantly under sparse-view conditions, especially in unseen regions where no ground-truth observations are available. While recent advances in diffusion models have demonstrated strong capabilities in synthesizing novel views from limited inputs, their outputs often suffer from visual artifacts and lack 3D consistency, posing challenges for reliable mesh optimization. In this paper, we propose a novel framework that leverages diffusion models to enhance sparse-view mesh reconstruction in a principled and reliable manner. To address the instability of diffusion outputs, we propose a Consensus Diffusion Module that filters unreliable generations via interquartile range (IQR) analysis and performs variance-aware image fusion to produce robust pseudo-supervision. Building on this, we design an online reinforcement learning strategy based on the Upper Confidence Bound (UCB) to adaptively select the most informative viewpoints for enhancement, guided by diffusion loss. Finally, the fused images are used to jointly supervise a NeRF-based model alongside sparse-view ground truth, ensuring consistency across both geometry and appearance. Extensive experiments demonstrate that our method achieves significant improvements in both geometric quality and rendering quality.

Problem

• Enhances sparse-view mesh reconstruction reliability
• Filters unstable diffusion outputs via IQR analysis
• Adaptively selects optimal viewpoints using reinforcement learning

Innovation

• Consensus Diffusion Module filters unreliable generations
• Reinforcement learning selects informative viewpoints adaptively
• NeRF-based model ensures geometry and appearance consistency