RbFT-Net: Rectify-Before-Fuse Temporal Radar Anchors for 4D Radar-Camera Depth Completion

2026-08-13

AI Summary

This work addresses the challenges of sparse, cluttered, and multipath-corrupted 4D millimeter-wave radar data, along with temporal misalignment, in dense depth completion from multi-frame radar–camera fusion. To this end, the authors propose RbFT-Net, an end-to-end “correct-then-fuse” framework that treats multi-frame radar echoes as noisy temporal anchors. An image-conditioned correction module jointly refines their image-plane coordinates and metric depths while estimating point-wise reliability scores. High-quality anchors are then selectively propagated for multimodal fusion based on these reliability estimates. The method innovatively integrates an image-guided radar correction mechanism and a reliability-aware propagation strategy to effectively mitigate noise contamination. Experiments demonstrate that RbFT-Net significantly outperforms existing radar–camera fusion approaches on both the ZJU-4DRadarCam benchmark and a newly collected 4D radar–camera–LiDAR dataset, achieving performance comparable to plug-in methods that leverage monocular depth priors.

Abstract

Dense metric depth prediction from cameras and millimeter-wave radar offers a cost-effective sensing solution for autonomous systems. However, radar measurements are inherently sparse and susceptible to clutter, multipath reflections, and projection errors. While aggregating multiple radar frames provides denser metric cues, it also introduces temporal misalignment and dynamic-object interference. Directly propagating such unreliable measurements can therefore corrupt large regions of the predicted depth map. To address this issue, we propose RbFT-Net, an end-to-end rectify-before-fuse framework for multi-frame 4D radar-camera depth completion. Rather than assuming accumulated radar returns to be accurate, RbFT-Net treats them as noisy temporal anchor candidates. An image-conditioned rectification module jointly corrects their image-plane locations and metric depths while estimating pointwise reliability. The rectified anchors are then selectively propagated before high-level multi-modal fusion, suppressing the influence of unreliable measurements. Experiments on ZJU-4DRadarCam and a newly collected 4D radar-camera-LiDAR dataset show that RbFT-Net consistently outperforms the evaluated independent radar-camera methods and remains competitive with plug-in pipelines using auxiliary monocular depth models. Cross-platform evaluation and component analyses further support the effectiveness of the proposed rectification and reliability-aware propagation strategy.

Problem

• 4D radar
• depth completion
• radar-camera fusion
• temporal misalignment
• sparse measurements

Innovation

• rectify-before-fuse
• temporal radar anchors
• 4D radar-camera depth completion
• reliability-aware propagation
• image-conditioned rectification