ObjectGS: Object-aware Scene Reconstruction and Scene Understanding via Gaussian Splatting

2025-07-21

AI Summary

3D Gaussian Splatting achieves high-fidelity reconstruction and real-time novel-view synthesis but lacks object-level semantic understanding. To address this, we propose an object-centric 3D Gaussian splatting framework: each scene object is modeled via a unique ID-anchored representation; a 1D object ID embedding and classification loss are introduced to enforce semantic consistency; and neural Gaussian rendering is integrated with dynamic anchor growth/pruning and shared one-hot ID encoding to enable precise object-aware rendering without compromising visual quality. Our method outperforms existing approaches on open-vocabulary segmentation and panoptic segmentation benchmarks. Moreover, it supports high-quality mesh extraction and interactive object editing. To our knowledge, this is the first work to unify high-fidelity radiance field reconstruction with rigorous object-level semantic understanding in a single differentiable framework.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

3D Gaussian Splatting is renowned for its high-fidelity reconstructions and real-time novel view synthesis, yet its lack of semantic understanding limits object-level perception. In this work, we propose ObjectGS, an object-aware framework that unifies 3D scene reconstruction with semantic understanding. Instead of treating the scene as a unified whole, ObjectGS models individual objects as local anchors that generate neural Gaussians and share object IDs, enabling precise object-level reconstruction. During training, we dynamically grow or prune these anchors and optimize their features, while a one-hot ID encoding with a classification loss enforces clear semantic constraints. We show through extensive experiments that ObjectGS not only outperforms state-of-the-art methods on open-vocabulary and panoptic segmentation tasks, but also integrates seamlessly with applications like mesh extraction and scene editing. Project page: https://ruijiezhu94.github.io/ObjectGS_page

Problem

• Lack of semantic understanding in 3D Gaussian Splatting
• Need for object-level perception in scene reconstruction
• Integration of semantic understanding with 3D scene reconstruction

Innovation

• Object-aware framework with semantic understanding
• Dynamic anchor growth and feature optimization
• One-hot ID encoding for semantic constraints