NerfBaselines: Consistent and Reproducible Evaluation of Novel View Synthesis Methods

2024-06-25

arXiv.org

AI Summary

New view synthesis evaluation suffers from three major challenges: inconsistent evaluation protocols, poor reproducibility, and limited cross-scene generalizability—leading to unreliable state-of-the-art tracking and questionable quantitative comparability. To address these issues, we propose the first standardized evaluation framework for neural radiance fields (NeRF) and 3D Gaussian splatting (3DGS). Our framework (1) establishes a cross-method consistent evaluation protocol; (2) enables one-click reproduction of multiple algorithms with CI-driven automated result validation; and (3) provides an interactive web platform for transparent, horizontal benchmarking across eight+ methods on five+ standard scenes. Implemented in Python/PyTorch, it unifies data loading, rendering, and evaluation pipelines. Experiments reproduce over ten mainstream methods and significantly reduce evaluation variance across multiple datasets, establishing a robust, reproducible benchmark for new view synthesis research.

Technology Category

• Computer Vision: 3D Computer Vision
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Consent frameworks and practices on the web

Abstract

Novel view synthesis is an important problem with many applications, including AR/VR, gaming, and simulations for robotics. With the recent rapid development of Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS) methods, it is becoming difficult to keep track of the current state of the art (SoTA) due to methods using different evaluation protocols, codebases being difficult to install and use, and methods not generalizing well to novel 3D scenes. Our experiments support this claim by showing that tiny differences in evaluation protocols of various methods can lead to inconsistent reported metrics. To address these issues, we propose a framework called NerfBaselines, which simplifies the installation of various methods, provides consistent benchmarking tools, and ensures reproducibility. We validate our implementation experimentally by reproducing numbers reported in the original papers. To further improve the accessibility, we release a web platform where commonly used methods are compared on standard benchmarks. Web: https://jkulhanek.com/nerfbaselines

Problem

• Addressing inconsistent evaluation protocols in novel view synthesis
• Ensuring reproducibility and comparability of quantitative results
• Simplifying installation and usage of diverse NVS methods

Innovation

• Framework provides consistent benchmarking tools
• Ensures reproducibility and simplifies method usage
• Releases web platform for method comparison