Multi4D: High-Fidelity Dynamic Gaussian Splatting via Multi-Level Competitive Allocation

2026-06-20

AI Summary

This work addresses the inherent trade-off in dynamic 3D Gaussian splatting between motion consistency and high-frequency detail preservation: deformation-based approaches maintain temporal correspondence but suffer from excessive smoothing, while 4D primitive methods retain fine details yet incur over-parameterization, identity ambiguity, and high storage costs. To resolve this, we propose Multi4D, a framework that adaptively allocates modeling capacity across three types of Gaussian primitives—static structure, persistent dynamic geometry, and transient appearance—via a multi-level competitive assignment mechanism. Coupled with shared rasterization and residual-driven optimization, Multi4D enables dynamic specialization without requiring predefined decomposition. Our method drastically reduces the number of dynamic primitives, achieving state-of-the-art rendering quality and real-time performance, while also setting new benchmarks in 4D semantic segmentation with an order-of-magnitude faster inference.

Abstract

Dynamic 3D Gaussian splatting faces a fundamental tension between motion consistency and visual fidelity. Deformation-based approaches preserve temporal correspondence but suffer from motion over-factorization, oversmoothing high-frequency dynamics. In contrast, 4D-primitive methods capture fine visual details yet incur temporal overparameterization, breaking object identity and leading to severe storage overhead. To resolve this, we introduce Multi4D, a framework for high-fidelity dynamic Gaussian Splatting based on multi-level competitive allocation. Instead of a monolithic representation, we distribute modeling capacity across three structured levels: static structure, persistent dynamic geometry, and transient appearance primitives. Through shared rasterization and residual-driven optimization, these levels dynamically compete to explain photometric error, enabling adaptive specialization without pre-assigned decomposition. This allocation preserves long-term motion consistency while capturing fine dynamic detail, achieving state-of-the-art rendering quality and real-time performance with significantly fewer dynamic primitives. Furthermore, because our representation explicitly tracks compact persistent Gaussians over time, semantic features can be embedded afterward, enabling Multi4D to achieve state-of-the-art 4D segmentation accuracy with an order-of-magnitude speedup. Project page: https://batfacewayne.github.io/Multi4D.io/

Problem

• Dynamic 3D Gaussian Splatting
• motion consistency
• visual fidelity
• temporal overparameterization
• motion over-factorization

Innovation

• Dynamic Gaussian Splatting
• Multi-Level Competitive Allocation
• 4D Scene Representation
• Motion Consistency
• High-Fidelity Rendering