SAGE: Semantic-Driven Adaptive Gaussian Splatting in Extended Reality

2025-03-20

AI Summary

To address the challenge of balancing visual fidelity and real-time performance in 3D Gaussian Splatting (3DGS) for XR applications, this paper proposes a semantics-driven adaptive Level-of-Detail (LOD) control method. It leverages lightweight real-time semantic segmentation (e.g., Mask2Former) to identify scene objects and enables object-level, fine-grained dynamic allocation of Gaussian ellipsoid density and geometric precision. Integrated with memory-aware rendering scheduling, the method significantly reduces GPU memory consumption and computational overhead while preserving target visual quality. To the best of our knowledge, this is the first approach to deeply integrate semantic understanding with 3DGS LOD control. Evaluated on XR devices, it achieves real-time, high-fidelity, low-latency 3D rendering: reducing memory usage by 32% and accelerating inference by 2.1× over baseline methods, while maintaining PSNR ≥ 28 dB.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

3D Gaussian Splatting (3DGS) has significantly improved the efficiency and realism of three-dimensional scene visualization in several applications, ranging from robotics to eXtended Reality (XR). This work presents SAGE (Semantic-Driven Adaptive Gaussian Splatting in Extended Reality), a novel framework designed to enhance the user experience by dynamically adapting the Level of Detail (LOD) of different 3DGS objects identified via a semantic segmentation. Experimental results demonstrate how SAGE effectively reduces memory and computational overhead while keeping a desired target visual quality, thus providing a powerful optimization for interactive XR applications.

Problem

• Enhance 3D scene visualization in XR applications
• Dynamically adjust Level of Detail via semantic segmentation
• Reduce memory and computational costs while maintaining visual quality

Innovation

• Semantic-driven adaptive 3D Gaussian splatting
• Dynamic Level of Detail (LOD) adjustment
• Optimized memory and computational efficiency