Improving Multi-View Reconstruction via Texture-Guided Gaussian-Mesh Joint Optimization

2025-11-06

AI Summary

In multi-view reconstruction, geometric accuracy (MVS) and appearance fidelity (NVS) are traditionally optimized in decoupled manners, hindering their joint improvement. Method: We propose a texture-guided Gaussian-mesh co-optimization framework, where differentiable Gaussian rendering jointly optimizes mesh vertex positions, face topology, and vertex colors under photometric consistency, normal/depth geometric regularization, and texture-aware priors. Contribution/Results: To our knowledge, this is the first method enabling end-to-end unified modeling of geometry and appearance, eliminating error accumulation inherent in conventional two-stage pipelines. Experiments demonstrate superior balance between reconstruction accuracy and visual realism, significantly enhancing quality and controllability in downstream editing tasks—including relighting and non-rigid deformation.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Search and Optimization: Non-convex Optimization

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Reconstructing real-world objects from multi-view images is essential for applications in 3D editing, AR/VR, and digital content creation. Existing methods typically prioritize either geometric accuracy (Multi-View Stereo) or photorealistic rendering (Novel View Synthesis), often decoupling geometry and appearance optimization, which hinders downstream editing tasks. This paper advocates an unified treatment on geometry and appearance optimization for seamless Gaussian-mesh joint optimization. More specifically, we propose a novel framework that simultaneously optimizes mesh geometry (vertex positions and faces) and vertex colors via Gaussian-guided mesh differentiable rendering, leveraging photometric consistency from input images and geometric regularization from normal and depth maps. The obtained high-quality 3D reconstruction can be further exploit in down-stream editing tasks, such as relighting and shape deformation. The code will be publicly available upon acceptance.

Problem

• Unifies geometry and appearance optimization for 3D reconstruction
• Simultaneously optimizes mesh geometry and vertex colors via Gaussian rendering
• Enables downstream editing tasks like relighting and shape deformation

Innovation

• Unified Gaussian-mesh joint optimization framework
• Simultaneously optimizes mesh geometry and vertex colors
• Leverages photometric consistency and geometric regularization