RadarTwin: Scene-Specific mmWave Radar Simulation and Learning for Mobile Indoor Perception

2026-06-24

AI Summary

This work addresses the challenge of limited generalization in millimeter-wave radar perception due to scarce real-world data across novel objects, environments, and trajectories. The authors propose a scene-customized simulation framework that requires no real radar data: leveraging 3D reconstructions of target scenes and surface material inference via vision-language models, it generates high-fidelity FMCW radar signals through physically accurate ray tracing and multipath modeling. This approach enables, for the first time, the synthesis of realistic radar training data prior to deployment, demonstrating strong correspondence between simulated and real radar responses in terms of shape and material characteristics. Using only synthetic data, the method achieves an object recognition accuracy 2.5 times higher than random guessing; with minimal real labeled data, it attains 95.3% accuracy on a 12-class object recognition task.

Abstract

Millimeter-wave (mmWave) radar perception is limited by data scarcity: models trained on existing radar datasets fail to generalize to new objects, environments, and sensing trajectories. We present RadarTwin, a framework for generating deployment-specific radar training data before real data collection. Given a 3D reconstruction of a target space (phone LiDAR, robot-mounted sensing, or RGB-to-3D), RadarTwin uses a vision-language model to infer radar-relevant surface materials and a physics-based ray tracer to synthesize raw frequency-modulated continuous-wave (FMCW) radar measurements with multi-bounce propagation. To study what transfers from simulation to reality, we collect a paired real-simulated dataset spanning household objects, material classes, distances, rotations, translations, and mobile sensing trajectories. We show that simulated and real radar share the same object-discriminative shape and material features, and that modeling the environment's multipath is essential to matching real measurements. A representation trained on simulation alone recognizes real objects at 2.5 times chance with no real radar labels, and a few labeled examples raise this to 95.3% on a 12-way recognition task. RadarTwin enables training radar perception for a new space before any real radar data is collected there.

Problem

• mmWave radar
• data scarcity
• generalization
• indoor perception
• simulation-to-reality

Innovation

• mmWave radar simulation
• scene-specific data generation
• physics-based ray tracing
• multipath modeling
• simulation-to-reality transfer