Next Best View Selections for Semantic and Dynamic 3D Gaussian Splatting

2025-12-27

AI Summary

To address viewpoint redundancy and information inefficiency in multi-camera dynamic scene modeling, this paper proposes an active Next Best View (NBV) selection method grounded in the Fisher Information Criterion (FIC), framing viewpoint selection as a joint semantic-dynamic active learning optimization problem. We introduce, for the first time, the use of Fisher information to quantify the information gain of candidate viewpoints for training both semantic 3D Gaussian splatting parameters and differentiable dynamic deformation networks—enabling synergistic optimization of semantic understanding and motion modeling. Compared to conventional heuristic or random frame selection, our approach achieves significant performance gains on large-scale static and dynamic datasets: +1.8 dB in PSNR and +4.2% in semantic segmentation mIoU. These results validate its dual advantages in rendering fidelity and semantic consistency.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Multi-instance/Multi-view Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Understanding semantics and dynamics has been crucial for embodied agents in various tasks. Both tasks have much more data redundancy than the static scene understanding task. We formulate the view selection problem as an active learning problem, where the goal is to prioritize frames that provide the greatest information gain for model training. To this end, we propose an active learning algorithm with Fisher Information that quantifies the informativeness of candidate views with respect to both semantic Gaussian parameters and deformation networks. This formulation allows our method to jointly handle semantic reasoning and dynamic scene modeling, providing a principled alternative to heuristic or random strategies. We evaluate our method on large-scale static images and dynamic video datasets by selecting informative frames from multi-camera setups. Experimental results demonstrate that our approach consistently improves rendering quality and semantic segmentation performance, outperforming baseline methods based on random selection and uncertainty-based heuristics.

Problem

• Selects informative frames for semantic and dynamic 3D Gaussian splatting
• Formulates view selection as an active learning problem using Fisher Information
• Improves rendering quality and semantic segmentation over heuristic baselines

Innovation

• Active learning with Fisher Information for view selection
• Quantifies informativeness of semantic and dynamic parameters
• Jointly handles semantic reasoning and dynamic scene modeling