FunFact: Building Probabilistic Functional 3D Scene Graphs via Factor-Graph Reasoning

2026-04-04

AI Summary

Existing approaches model pairwise object relationships in isolation, struggling to capture scene-level functional dependencies necessary for resolving semantic ambiguities. This work proposes a novel framework that constructs an object- and part-centric 3D map from posed RGB-D images, leverages foundation models to generate open-vocabulary functional relationship candidates, and—uniquely—integrates factor graph reasoning with commonsense priors derived from large language models and geometric constraints to enable joint probabilistic inference over full-scene functional relationships. The method substantially improves relationship recall and confidence calibration, demonstrating strong empirical performance on SceneFun3D, FunGraph3D, and FunThor, a newly introduced benchmark dataset.

Abstract

Recent work in 3D scene understanding is moving beyond purely spatial analysis toward functional scene understanding. However, existing methods often consider functional relationships between object pairs in isolation, failing to capture the scene-wide interdependence that humans use to resolve ambiguity. We introduce FunFact, a framework for constructing probabilistic open-vocabulary functional 3D scene graphs from posed RGB-D images. FunFact first builds an object- and part-centric 3D map and uses foundation models to propose semantically plausible functional relations. These candidates are converted into factor graph variables and constrained by both LLM-derived common-sense priors and geometric priors. This formulation enables joint probabilistic inference over all functional edges and their marginals, yielding substantially better calibrated confidence scores. To benchmark this setting, we introduce FunThor, a synthetic dataset based on AI2-THOR with part-level geometry and rule-based functional annotations. Experiments on SceneFun3D, FunGraph3D, and FunThor show that FunFact improves node and relation discovery recall and significantly reduces calibration error for ambiguous relations, highlighting the benefits of holistic probabilistic modeling for functional scene understanding. See our project page at https://funfact-scenegraph.github.io/

Problem

• functional scene understanding
• 3D scene graphs
• scene-wide interdependence
• ambiguous relations
• probabilistic modeling

Innovation

• probabilistic scene graph
• factor-graph reasoning
• functional 3D understanding
• open-vocabulary relations
• LLM-guided priors