NeuRadar: Neural Radiance Fields for Automotive Radar Point Clouds

2025-04-01

AI Summary

To address the challenge of jointly modeling multi-modal sensor data (radar, camera, LiDAR) under adverse weather and varying illumination conditions in autonomous driving simulation, this work pioneers the extension of Neural Radiance Fields (NeRF) to vehicular millimeter-wave radar point clouds. We propose a radar-NeRF representation that synergistically integrates deterministic and probabilistic modeling. By unifying a DETR-style set-based detection framework with NeRF’s geometric priors, our method achieves cross-modal consistent joint rendering. It significantly improves modeling fidelity for radar’s inherent stochasticity and enhances cross-sensor generalization. We demonstrate high-fidelity reconstructions on two real-world automotive radar datasets, establishing the first radar-NeRF simulation baseline. To foster community advancement, we publicly release the ZOD radar sequence dataset and the complete NeuRadar codebase.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Radar is an important sensor for autonomous driving (AD) systems due to its robustness to adverse weather and different lighting conditions. Novel view synthesis using neural radiance fields (NeRFs) has recently received considerable attention in AD due to its potential to enable efficient testing and validation but remains unexplored for radar point clouds. In this paper, we present NeuRadar, a NeRF-based model that jointly generates radar point clouds, camera images, and lidar point clouds. We explore set-based object detection methods such as DETR, and propose an encoder-based solution grounded in the NeRF geometry for improved generalizability. We propose both a deterministic and a probabilistic point cloud representation to accurately model the radar behavior, with the latter being able to capture radar's stochastic behavior. We achieve realistic reconstruction results for two automotive datasets, establishing a baseline for NeRF-based radar point cloud simulation models. In addition, we release radar data for ZOD's Sequences and Drives to enable further research in this field. To encourage further development of radar NeRFs, we release the source code for NeuRadar.

Problem

• NeRF-based synthesis of radar point clouds
• Joint generation of radar, camera, lidar data
• Modeling radar's stochastic behavior accurately

Innovation

• NeRF-based model for radar and sensor data
• Deterministic and probabilistic radar representations
• Encoder-based solution for improved generalizability