PlanarNeRF: Online Learning of Planar Primitives with Neural Radiance Fields

2023-12-30

arXiv.org

AI Summary

Existing methods struggle to reconstruct spatially complete, dense 3D planar primitives from visual data, often resorting to 2D segmentation or oversimplified 3D modeling; they remain performance-limited despite heavy reliance on annotated data. This paper proposes an online-learning-enabled framework for dense 3D planar reconstruction. It introduces the first online planar detection method jointly leveraging appearance and geometric priors; designs a lightweight, differentiable plane-fitting module; and constructs a global memory bank with dynamic updating to ensure inter-frame consistency. Built upon Neural Radiance Fields (NeRF), the framework supports both sparse supervision and self-supervised training. Extensive experiments across diverse scenes demonstrate substantial improvements over state-of-the-art methods—achieving higher planar detection accuracy and significantly enhanced training efficiency.

Technology Category

• Computer Vision: 3D Computer Vision
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Machine Learning: Learning with Manifolds

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Identifying spatially complete planar primitives from visual data is a crucial task in computer vision. Prior methods are largely restricted to either 2D segment recovery or simplifying 3D structures, even with extensive plane annotations. We present PlanarNeRF, a novel framework capable of detecting dense 3D planes through online learning. Drawing upon the neural field representation, PlanarNeRF brings three major contributions. First, it enhances 3D plane detection with concurrent appearance and geometry knowledge. Second, a lightweight plane fitting module is proposed to estimate plane parameters. Third, a novel global memory bank structure with an update mechanism is introduced, ensuring consistent cross-frame correspondence. The flexible architecture of PlanarNeRF allows it to function in both 2D-supervised and self-supervised solutions, in each of which it can effectively learn from sparse training signals, significantly improving training efficiency. Through extensive experiments, we demonstrate the effectiveness of PlanarNeRF in various scenarios and remarkable improvement over existing works.

Problem

• Detects dense 3D planes using online learning
• Improves 3D plane detection with appearance and geometry
• Enhances training efficiency with sparse signals

Innovation

• Online learning for dense 3D plane detection
• Lightweight module for plane parameter estimation
• Global memory bank ensures cross-frame consistency