PUF: Plug-and-Play Uncertainty-Aware Fusion for Online 3D Scene Graph Generation

2026-07-08

AI Summary

Existing online 3D scene graph generation methods overlook uncertainties inherent in observations, 2D models, and 3D representations, leading to overly deterministic fusion processes. This work proposes a plug-and-play, training-free, uncertainty-aware fusion framework that, for the first time, integrates explicit uncertainty modeling into online 3D scene graph construction. The approach jointly models semantic and spatial factors through probabilistic likelihoods to infer node associations, accumulates categorical and relational evidence using Dirichlet-based evidential reasoning, and replaces conventional binary gating with probabilistic association. Compatible with diverse 3D representations—including Gaussian and voxel-based formats—the method achieves state-of-the-art performance on both 3DSSG and ReplicaSSG benchmarks while maintaining real-time inference speed.

Abstract

Online 3D scene graph generation builds a persistent, structured representation of a scene by incrementally fusing 2D observations into a global 3D graph. Existing online methods treat this fusion as a fully deterministic pipeline, where we identify three sources of uncertainty that are overlooked: observation, 2D model, and 3D representation. We propose PUF: a Plug-and-play, Uncertainty-aware, and training-free Fusion framework. Scene graph node association is reformulated as a probabilistic likelihood over semantic and spatial factors, replacing binary accept/reject gates. Dirichlet evidence accumulation distributes class and relationship evidence across plausible candidates proportional to association likelihood. An optional class-conditional prior completes edges for sparsely or never co-observed object pairs. We instantiate PUF with both a 3D Gaussian and a 3D voxel backend and observe consistent improvements, demonstrating its ability to generalize across different representations. Experiments on the 3DSSG and ReplicaSSG benchmarks show that our method substantially outperforms existing approaches while maintaining real-time latency. These results establish uncertainty-aware fusion as a principled and effective paradigm for online 3D scene understanding. The source code is publicly available at https://github.com/yyyyangyi/PUF.

Problem

• 3D scene graph
• online fusion
• uncertainty
• scene understanding
• probabilistic modeling

Innovation

• uncertainty-aware fusion
• online 3D scene graph
• probabilistic association
• Dirichlet evidence accumulation
• plug-and-play framework