Streaming Real-Time Rendered Scenes as 3D Gaussians

2026-04-03

AI Summary

This work addresses the limitations of traditional cloud rendering, which relies on 2D video streams and struggles to support flexible viewpoint switching and efficient latency compensation. The paper introduces 3D Gaussian Splatting (3DGS) into real-time cloud rendering for the first time, proposing a streaming framework in which the server dynamically constructs and continuously optimizes a 3DGS scene, transmitting it to clients via full snapshots and incremental updates. Clients locally render arbitrary viewpoints from the received 3D representation and further support relighting and rigid-body dynamics. Implemented in Unity, the system enables multiple users to share the same scene model. Experiments demonstrate that, compared to conventional image-warping approaches, the proposed method significantly enhances viewpoint freedom and improves server-side resource reuse efficiency.

Abstract

Cloud rendering is widely used in gaming and XR to overcome limited client-side GPU resources and to support heterogeneous devices. Existing systems typically deliver the rendered scene as a 2D video stream, which tightly couples the transmitted content to the server-rendered viewpoint and limits latency compensation to image-space reprojection or warping. In this paper, we investigate an alternative approach based on streaming a live 3D Gaussian Splatting (3DGS) scene representation instead of only rendered video. We present a Unity-based prototype in which a server constructs and continuously optimizes a 3DGS model from real-time rendered reference views, while streaming the evolving representation to remote clients using full model snapshots and incremental updates supporting relighting and rigid object dynamics. The clients reconstruct the streamed Gaussian model locally and render their current viewpoint from the received representation. This approach aims to improve viewpoint flexibility for latency compensation and to better amortize server-side scene modeling across multiple users than per-user rendering and video streaming. We describe the system design, evaluate it, and compare it with conventional image warping.

Problem

• cloud rendering
• 3D Gaussian Splatting
• viewpoint flexibility
• latency compensation
• real-time streaming

Innovation

• 3D Gaussian Splatting
• cloud rendering
• streaming 3D representation
• viewpoint flexibility
• real-time rendering