A self-supervised cyclic neural-analytic approach for novel view synthesis and 3D reconstruction

2025-03-05

AI Summary

To address poor generalization in novel-view synthesis and 3D reconstruction from sparse views for autonomous UAV navigation, this paper proposes a neural-analytic co-designed self-supervised cycle framework. The method integrates a Transformer-based image reconstruction network, differentiable geometric priors, and a NeRF variant, incorporating self-supervised cycle-consistency constraints while jointly optimizing analytic depth and surface normals. It operates without annotated data, enabling closed-loop self-supervised training. The framework significantly improves generalization to completely unseen camera poses and undersampled regions. Evaluated on complex outdoor scenes, it achieves a 2.1 dB PSNR gain in novel-view RGB rendering and a 37% reduction in mesh Chamfer distance. To our knowledge, this is the first approach to simultaneously achieve state-of-the-art rendering quality and geometric accuracy.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Unsupervised & Self-Supervised Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Generating novel views from recorded videos is crucial for enabling autonomous UAV navigation. Recent advancements in neural rendering have facilitated the rapid development of methods capable of rendering new trajectories. However, these methods often fail to generalize well to regions far from the training data without an optimized flight path, leading to suboptimal reconstructions. We propose a self-supervised cyclic neural-analytic pipeline that combines high-quality neural rendering outputs with precise geometric insights from analytical methods. Our solution improves RGB and mesh reconstructions for novel view synthesis, especially in undersampled areas and regions that are completely different from the training dataset. We use an effective transformer-based architecture for image reconstruction to refine and adapt the synthesis process, enabling effective handling of novel, unseen poses without relying on extensive labeled datasets. Our findings demonstrate substantial improvements in rendering views of novel and also 3D reconstruction, which to the best of our knowledge is a first, setting a new standard for autonomous navigation in complex outdoor environments.

Problem

• Improves novel view synthesis for autonomous UAV navigation.
• Enhances 3D reconstruction in undersampled and unseen regions.
• Combines neural rendering with geometric insights for better accuracy.

Innovation

• Self-supervised cyclic neural-analytic pipeline
• Transformer-based architecture for image reconstruction
• Combines neural rendering with geometric insights