Simulating Automotive Radar with Lidar and Camera Inputs

2025-03-11

AI Summary

To address the scarcity of high-fidelity 4D radar data—comprising elevation angle, azimuth angle, range, Doppler velocity, and signal amplitude—from low-cost mmWave radar, which hinders robustness research in autonomous driving, this paper proposes the first physics-informed radar synthesis framework jointly driven by geometry and appearance. We introduce a dual-network architecture: DIS-Net models spatial distribution and cardinality of radar returns, while RSS-Net models signal amplitude; both networks are trained end-to-end using RGB images, LiDAR point clouds, and ego-motion information. Our method generates photorealistic 4D radar data on three commercial radar datasets. Evaluations demonstrate substantial improvements in radar-based object detection: models trained solely on synthetic data surpass those trained exclusively on real data, establishing a new state-of-the-art baseline.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Low-cost millimeter automotive radar has received more and more attention due to its ability to handle adverse weather and lighting conditions in autonomous driving. However, the lack of quality datasets hinders research and development. We report a new method that is able to simulate 4D millimeter wave radar signals including pitch, yaw, range, and Doppler velocity along with radar signal strength (RSS) using camera image, light detection and ranging (lidar) point cloud, and ego-velocity. The method is based on two new neural networks: 1) DIS-Net, which estimates the spatial distribution and number of radar signals, and 2) RSS-Net, which predicts the RSS of the signal based on appearance and geometric information. We have implemented and tested our method using open datasets from 3 different models of commercial automotive radar. The experimental results show that our method can successfully generate high-fidelity radar signals. Moreover, we have trained a popular object detection neural network with data augmented by our synthesized radar. The network outperforms the counterpart trained only on raw radar data, a promising result to facilitate future radar-based research and development.

Problem

• Simulate 4D radar signals using camera, lidar, and ego-velocity inputs.
• Address lack of quality datasets for automotive radar research.
• Enhance object detection with synthesized radar data augmentation.

Innovation

• Simulates 4D radar signals using camera, lidar, and ego-velocity
• Introduces DIS-Net and RSS-Net neural networks for radar simulation
• Enhances object detection with synthesized radar data augmentation