DLGStream: Dynamic Language-embedded Guassian Splatting for Open-vocabulary Enabled Free-viewpoint Video Streaming

2026-06-27

AI Summary

Existing 3D Gaussian splatting methods lack language-driven interactivity and editability in free-viewpoint video and struggle to simultaneously achieve low-bandwidth, high-frame-rate streaming. This work proposes a dynamic language-embedded Gaussian splatting representation that decouples color and language feature optimization via a dual-opacity mechanism. It integrates time-varying CLIP language embeddings and an interpolation-based deformation field to reduce temporal redundancy, while incorporating open-vocabulary distillation for semantic awareness. The resulting framework enables high-FPS free-viewpoint video streaming with an average frame size of only 43 KB, supporting both high-fidelity reconstruction and open-vocabulary scene editing. This significantly enhances spatial intelligence and interactive capabilities in 4D environments.

Abstract

3D Gaussian Splatting~(3DGS) has emerged as a promising paradigm for reconstructing streamable free-viewpoint video~(FVV) from multi-view videos. However, 3DGS-based FVVs typically lack user interaction and editing capabilities, which diminishes the immersive experience. Recent research has integrated language features from CLIP into 3DGS via distillation, enabling open-vocabulary queries and supporting many downstream applications. Nevertheless, the stringent requirements of FVV, low frame size and high FPS, make current language Gaussian representations unsuitable for language-embedded FVV. In this paper, we propose DLGStream, a novel language-embedded FVV representation that streams time-varying language features alongside Gaussian attributes to support 4D environment interaction, scene editing, and spatial intelligence. Specifically, we propose a dual-opacity dynamic language Gaussian representation, which maintains two opacity attributes for color and language features to deal with performance degradation that occurs when colors and features are jointly optimized. Furthermore, we introduce an interpolation-based deformation field to reduce temporal redundancy. This deformation field can also be used for 4D frame interpolation, boosting FVV sequences from low to high FPS. Experimental results demonstrate that DLGStream achieves superior performance in both on open-vocabulary segmentation and reconstruction quality with an average frame size of merely 43 KB. The code is available on \href{https://github.com/kkkzh/DLGStream}{https://github.com/kkkzh/DLGStream}.

Problem

• Free-viewpoint Video
• 3D Gaussian Splatting
• Open-vocabulary
• Language-embedded Representation
• Interactive Streaming

Innovation

• Dynamic Language-embedded Gaussian Splatting
• Open-vocabulary Interaction
• Dual-opacity Representation
• 4D Deformation Field
• Free-viewpoint Video Streaming