Streaming Gaussian Encoding for 4D Panoptic Occupancy Tracking

2026-06-29

AI Summary

Existing camera-based 4D panoptic occupancy tracking (4D-POT) methods suffer from insufficient temporal geometric consistency, particularly in occluded and static regions. This work proposes a streaming Gaussian encoder that introduces persistent Gaussian representations into 4D-POT for the first time. It maintains a voxelized scene representation through latent Gaussian query propagation, ego-motion compensation, and a confidence-guided update mechanism. Depth supervision is leveraged to shape Gaussian opacities as proxies for visibility, and multi-frame warm-up training is employed to enhance temporal consistency. The method achieves a new state-of-the-art among camera-based 4D-POT approaches on both Occ3D-extended nuScenes and Waymo benchmarks, significantly improving tracking consistency while incurring minimal computational overhead and remaining compatible with existing mask-based pipelines.

Abstract

Camera-based 4D panoptic occupancy tracking (4D-POT) is a promising paradigm for holistic scene understanding from multi-view imagery, enabling joint reasoning about geometry, semantics, and object identities across time. Recent mask-based pipelines achieve strong performance by propagating instance queries across frames. However, their underlying volumetric representations are typically recomputed at each timestep, limiting geometric temporal consistency, particularly under occlusion and for static scene elements. To address this limitation, we propose a streaming Gaussian encoder that maintains a persistent volumetric scene representation for 4D-POT. Our method models the scene as a fixed-size set of latent Gaussian queries that are propagated via ego-motion compensation and refreshed under a confidence-guided budget constraint. Crucially, we shape Gaussian opacities through depth-based supervision to serve as proxy for visibility, enabling confidence to accumulate as a temporally aggregated measure of persistent scene support. Together with a warmup-based multi-frame training strategy, this yields representation-level temporal coherence beyond decoder-only tracking. Extensive experiments on Occ3D-extended nuScenes and Waymo establish a new state-of-the-art for camera-based 4D-POT, improving tracking consistency with negligible computational overhead while remaining fully compatible with existing mask-based pipelines. We provide code and models at https://sge.cs.uni-freiburg.de.

Problem

• 4D panoptic occupancy tracking
• temporal consistency
• volumetric representation
• occlusion
• static scene elements

Innovation

• streaming Gaussian encoding
• 4D panoptic occupancy tracking
• temporal consistency
• volumetric representation
• confidence-guided propagation