4DGC: Rate-Aware 4D Gaussian Compression for Efficient Streamable Free-Viewpoint Video

2025-03-24

AI Summary

To address the limitations of dynamic 3D Gaussian Splatting (3DGS) in free-viewpoint video (FVV) compression—specifically, its neglect of explicit motion modeling and rate-distortion (RD) joint optimization—this work proposes the first motion-aware 4D Gaussian dynamic representation framework with end-to-end RD-optimized compression. Methodologically, it innovatively integrates motion meshes with sparse compensation Gaussians for accurate temporal modeling; introduces differentiable quantization and a lightweight implicit entropy model to enable end-to-end RD optimization and variable-bitrate coding. Evaluated on multiple standard benchmarks, the method significantly reduces Gaussian count and storage footprint—by up to 62%—while consistently outperforming state-of-the-art (SOTA) approaches across the full RD curve. It achieves an unprecedented balance between high compression efficiency and high-fidelity reconstruction, advancing the frontier of neural FVV compression.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Learning on the Edge & Model Compression
• Data Mining & Knowledge Management: Data Compression

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

3D Gaussian Splatting (3DGS) has substantial potential for enabling photorealistic Free-Viewpoint Video (FVV) experiences. However, the vast number of Gaussians and their associated attributes poses significant challenges for storage and transmission. Existing methods typically handle dynamic 3DGS representation and compression separately, neglecting motion information and the rate-distortion (RD) trade-off during training, leading to performance degradation and increased model redundancy. To address this gap, we propose 4DGC, a novel rate-aware 4D Gaussian compression framework that significantly reduces storage size while maintaining superior RD performance for FVV. Specifically, 4DGC introduces a motion-aware dynamic Gaussian representation that utilizes a compact motion grid combined with sparse compensated Gaussians to exploit inter-frame similarities. This representation effectively handles large motions, preserving quality and reducing temporal redundancy. Furthermore, we present an end-to-end compression scheme that employs differentiable quantization and a tiny implicit entropy model to compress the motion grid and compensated Gaussians efficiently. The entire framework is jointly optimized using a rate-distortion trade-off. Extensive experiments demonstrate that 4DGC supports variable bitrates and consistently outperforms existing methods in RD performance across multiple datasets.

Problem

• Compress dynamic 3D Gaussians for efficient FVV streaming
• Reduce storage and transmission costs while maintaining quality
• Optimize rate-distortion trade-off in 4D Gaussian representation

Innovation

• Motion-aware dynamic Gaussian representation with compact grid
• End-to-end compression using differentiable quantization
• Joint optimization with rate-distortion trade-off