AG$^2$aussian: Anchor-Graph Structured Gaussian Splatting for Instance-Level 3D Scene Understanding and Editing

2025-08-03

AI Summary

To address the challenges of instance segmentation noise and disorganized Gaussian selection in semantic-aware 3D Gaussian Splatting (3DGS), this paper proposes an Anchor Graph-Structured Gaussian Lattice framework. Our method constructs a semantic anchor graph that explicitly organizes Gaussian primitives into a topology centered on learnable semantic anchors, enabling graph-based semantic feature propagation and instance-level alignment. By integrating differentiable rendering, graph neural networks, and semantic anchor modeling, the framework preserves real-time rendering efficiency while significantly improving instance segmentation accuracy and semantic consistency of the Gaussian distribution. Experiments demonstrate state-of-the-art performance across four tasks—click-based querying, text-driven editing, object removal, and physics-aware simulation—validating the framework’s effectiveness and generalizability for fine-grained, interactive 3D scene understanding and editing.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Sampling/Simulation-based Search
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search

Abstract

3D Gaussian Splatting (3DGS) has witnessed exponential adoption across diverse applications, driving a critical need for semantic-aware 3D Gaussian representations to enable scene understanding and editing tasks. Existing approaches typically attach semantic features to a collection of free Gaussians and distill the features via differentiable rendering, leading to noisy segmentation and a messy selection of Gaussians. In this paper, we introduce AG$^2$aussian, a novel framework that leverages an anchor-graph structure to organize semantic features and regulate Gaussian primitives. Our anchor-graph structure not only promotes compact and instance-aware Gaussian distributions, but also facilitates graph-based propagation, achieving a clean and accurate instance-level Gaussian selection. Extensive validation across four applications, i.e. interactive click-based query, open-vocabulary text-driven query, object removal editing, and physics simulation, demonstrates the advantages of our approach and its benefits to various applications. The experiments and ablation studies further evaluate the effectiveness of the key designs of our approach.

Problem

• Enables semantic-aware 3D Gaussian representations for scene understanding
• Reduces noisy segmentation and messy Gaussian selection in 3DGS
• Organizes semantic features via anchor-graph for instance-level editing

Innovation

• Anchor-graph structured Gaussian splatting
• Instance-aware Gaussian distributions
• Graph-based propagation for clean selection