LapisGS: Layered Progressive 3D Gaussian Splatting for Adaptive Streaming

2024-08-27

arXiv.org

AI Summary

To address the challenge of efficiently streaming online 3D worlds using 3D Gaussian Splatting (3DGS) under bandwidth-constrained extended reality (XR) scenarios, this paper proposes a hierarchical 3D Gaussian Splatting representation. Our method introduces: (1) a novel hierarchical cumulative Gaussian structure enabling progressive decoding and rendering; and (2) dynamic opacity optimization coupled with an occupancy-map-guided sparsification scheduling mechanism, achieving continuous quality–bandwidth trade-offs. Experimental results demonstrate that, at only 23% of the original model size, our approach improves SSIM by 50.71% and reduces LPIPS by 286.53%, significantly enhancing bandwidth adaptability and visual fidelity. This work establishes a new paradigm for lightweight, adaptive 3D streaming.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Time-Series/Data Streams

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

The rise of Extended Reality (XR) requires efficient streaming of 3D online worlds, challenging current 3DGS representations to adapt to bandwidth-constrained environments. This paper proposes LapisGS, a layered 3DGS that supports adaptive streaming and progressive rendering. Our method constructs a layered structure for cumulative representation, incorporates dynamic opacity optimization to maintain visual fidelity, and utilizes occupancy maps to efficiently manage Gaussian splats. This proposed model offers a progressive representation supporting a continuous rendering quality adapted for bandwidth-aware streaming. Extensive experiments validate the effectiveness of our approach in balancing visual fidelity with the compactness of the model, with up to 50.71% improvement in SSIM, 286.53% improvement in LPIPS with 23% of the original model size, and shows its potential for bandwidth-adapted 3D streaming and rendering applications.

Problem

• Adaptive streaming for bandwidth-constrained XR environments
• Progressive rendering for continuous quality adaptation
• Efficient management of 3D Gaussian splats

Innovation

• Layered 3D Gaussian Splatting
• Dynamic opacity optimization
• Occupancy maps management