SERES: Semantic-aware neural reconstruction from sparse views

2025-08-23

AI Summary

To address radiometric blurring and geometric ambiguity arising from challenging feature matching in sparse multi-view image reconstruction, this paper proposes a semantic-enhanced neural implicit representation method. Our approach jointly optimizes a signed distance field (SDF) and a radiance field, incorporating semantic logits as a geometry-appearance co-constraint, and introduces geometric primitive mask regularization to mitigate shape and radiance uncertainty under sparse input. The method supports patch-level semantic supervision and can be seamlessly integrated—without architectural modification—into mainstream frameworks such as NeuS and Neuralangelo. On the DTU dataset, our method reduces Chamfer distance by 44% and 20% compared to SparseNeuS and VolRecon, respectively. When embedded as a plug-in into NeuS and Neuralangelo, it further decreases reconstruction error by 69% and 68%. These results demonstrate significant improvements in high-fidelity 3D semantic reconstruction from sparse views.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

We propose a semantic-aware neural reconstruction method to generate 3D high-fidelity models from sparse images. To tackle the challenge of severe radiance ambiguity caused by mismatched features in sparse input, we enrich neural implicit representations by adding patch-based semantic logits that are optimized together with the signed distance field and the radiance field. A novel regularization based on the geometric primitive masks is introduced to mitigate shape ambiguity. The performance of our approach has been verified in experimental evaluation. The average chamfer distances of our reconstruction on the DTU dataset can be reduced by 44% for SparseNeuS and 20% for VolRecon. When working as a plugin for those dense reconstruction baselines such as NeuS and Neuralangelo, the average error on the DTU dataset can be reduced by 69% and 68% respectively.

Problem

• Reconstructs 3D models from sparse image views
• Reduces radiance ambiguity via semantic-aware neural representations
• Mitigates shape ambiguity with geometric primitive regularization

Innovation

• Semantic-aware neural reconstruction from sparse views
• Patch-based semantic logits enrich neural representations
• Geometric primitive masks regularization reduces shape ambiguity