Toward a Low-Cost Perception System in Autonomous Vehicles: A Spectrum Learning Approach

2025-02-04

AI Summary

To address the challenges of sparsity, low resolution, and poor robustness in radar-derived depth maps for cost-effective autonomous driving—particularly in complex urban environments—this paper proposes a dense depth map generation method fusing 4D radar and RGB imagery. We innovatively introduce a pixel-wise positional encoding based on Bartlett spatial spectral estimation, establishing a radar-camera joint spectral learning framework that enables end-to-end generative modeling under a unified spatial-spectral representation. The method integrates 4D radar signal processing, spatial-spectral encoding, cross-modal alignment, and generative modeling, and employs unidirectional Chamfer distance for evaluation. Our approach achieves a 27.95% improvement over state-of-the-art methods on the UCD metric, significantly enhancing depth map density, spatial resolution, and environmental adaptability.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Multi-modal Vision
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

We present a cost-effective new approach for generating denser depth maps for Autonomous Driving (AD) and Autonomous Vehicles (AVs) by integrating the images obtained from deep neural network (DNN) 4D radar detectors with conventional camera RGB images. Our approach introduces a novel pixel positional encoding algorithm inspired by Bartlett's spatial spectrum estimation technique. This algorithm transforms both radar depth maps and RGB images into a unified pixel image subspace called the Spatial Spectrum, facilitating effective learning based on their similarities and differences. Our method effectively leverages high-resolution camera images to train radar depth map generative models, addressing the limitations of conventional radar detectors in complex vehicular environments, thus sharpening the radar output. We develop spectrum estimation algorithms tailored for radar depth maps and RGB images, a comprehensive training framework for data-driven generative models, and a camera-radar deployment scheme for AV operation. Our results demonstrate that our approach also outperforms the state-of-the-art (SOTA) by 27.95% in terms of Unidirectional Chamfer Distance (UCD).

Problem

• Cost-effective depth map generation for AVs
• Integration of radar and RGB images
• Novel pixel positional encoding algorithm

Innovation

• Integrates DNN radar with camera images
• Uses novel pixel positional encoding
• Develops tailored spectrum estimation algorithms