4DR360: State Reasoning for Joint 3D Detection and Occupancy Prediction in 4D Radar-Camera Full-Scene Perception

2026-07-10

AI Summary

Existing radar-camera fusion methods struggle to efficiently perform joint 3D object detection and semantic occupancy prediction, often lacking deep task-level synergy. This work proposes a cross-modal state reasoning framework that, for the first time, formulates semantic occupancy as a propagatable scene state. Within the bird’s-eye-view (BEV) space, intra-frame features are enhanced through state-guided refinement, while inter-frame state propagation is driven by Doppler information from 4D millimeter-wave radar, enabling comprehensive 360° scene perception. The approach establishes a unified radar-camera multi-task learning paradigm, achieving significant gains in accuracy, robustness, and efficiency on the extended ManTruckScenes and OmniHD-Scenes datasets. Furthermore, it introduces the first unified benchmark for jointly evaluating 3D detection and semantic occupancy prediction.

Abstract

Reliable autonomous driving requires full-scene perception that couples foreground objects with dense semantic layout. Recently, 4D millimeter-wave radar has emerged as a robust and affordable sensor, yet its sparse returns make radar-camera fusion necessary for comprehensive scene understanding. Existing radar-camera methods mainly optimize detection, while dual-task systems usually decode boxes and occupancy with limited interaction. To address this gap and advance radar-based multi-task learning, we propose \method, a 4D radar-camera framework for 360$^\circ$ full-scene perception, which models semantic occupancy as a persistent scene state rather than a terminal output. \method{} follows a cross-modal state reasoning paradigm, where the occupancy state is modeled and propagated through stages for coarse-to-fine feature aggregation. Specifically, State-guided BEV Enhancement (SBE) strengthens intra-frame BEV representation, while Doppler-guided Temporal Fusion (DTF) preserves state evidence over longer temporal horizons. Beyond the model, we further extend ManTruckScenes with satellite-map-based generated occupancy labels and pair it with OmniHD-Scenes in a unified cross-dataset detection-and-occupancy protocol. The resulting experiments cover accuracy, robustness, ablation, and efficiency under one radar-camera multi-task evaluation framework. Code and labels will be released upon acceptance.

Problem

• 4D radar-camera fusion
• 3D object detection
• semantic occupancy prediction
• full-scene perception
• multi-task learning

Innovation

• 4D radar-camera fusion
• semantic occupancy prediction
• state reasoning
• BEV representation
• multi-task perception