CRISP: A Spatiotemporal Camera-Radar Backbone for Driving via Forecasting-Based World-Model Pretraining

2026-07-05

AI Summary

This work addresses the limitation of existing camera-radar fusion methods, which rely on task-specific supervision and struggle to learn generalizable representations. The authors propose a self-supervised pretraining framework that constructs a spatiotemporally unified bird’s-eye-view (BEV) representation using only camera and radar inputs by predicting future LiDAR point clouds. Key innovations include an enhanced radar encoder, a radar-augmented temporal self-attention mechanism, and a multimodal feature renderer with a modality-aware gating scheme that effectively integrates radar range and Doppler information. Evaluated on nuScenes, the model significantly improves long-horizon point cloud prediction and demonstrates strong transferability across diverse downstream tasks—including 3D detection, tracking, online mapping, motion forecasting, future occupancy prediction, and planning—highlighting its potential as a foundational perception model for autonomous driving.

Abstract

Camera-radar (CR) fusion is a practical sensing configuration for autonomous driving, but existing models are typically trained with task-specific supervision, limiting reusable representation learning. We present CRISP, a spatiotemporal CR backbone pretrained through forecasting-based representation learning. Given historical multi-view images and radar sweeps, CRISP learns a unified bird's-eye-view (BEV) representation by predicting future LiDAR point clouds. LiDAR is used only as privileged supervision during pretraining; the deployed model requires only camera and radar. To make forecasting-based pretraining effective for CR fusion, CRISP introduces an enhanced radar encoder, radar-enhanced temporal self-attention, and multimodal feature rendering with modality innovation gating. These components inject radar range and Doppler cues into BEV temporal propagation and allow BEV tokens to selectively incorporate camera and radar evidence. Experiments on nuScenes show that CRISP improves long-horizon point cloud forecasting and transfers effectively to downstream tasks, including 3D detection, tracking, online mapping, motion forecasting, future occupancy prediction, and planning, suggesting that predictive CR pretraining is a promising path toward scalable driving representations under practical sensor configurations. The project website is https://umfieldrobotics.github.io/CRISP.

Problem

• camera-radar fusion
• representation learning
• autonomous driving
• sensor configuration
• predictive pretraining

Innovation

• camera-radar fusion
• forecasting-based pretraining
• bird's-eye-view representation
• privileged distillation
• multimodal perception