Semantically Annotated Multimodal Dataset for RF Interpretation and Prediction

2026-04-01

AI Summary

This work addresses a critical limitation in current radio frequency (RF) artificial intelligence: the scarcity of high-quality, real-world datasets that enable precise semantic association between RF signals and their physical environments, thereby hindering effective supervised learning. To bridge this gap, the authors present the first multimodal RF semantic dataset supporting bidirectional inference. It comprises synchronized RF measurements, high-resolution images, and LiDAR data collected across diverse indoor and outdoor scenes. Through high-precision spatiotemporal alignment and voxel-level semantic annotation, the dataset achieves fine-grained geometric and semantic correspondence between RF signals and environmental context. This enables both forward inference—predicting RF heatmaps from visual inputs—and inverse inference—recovering scene semantics from RF observations—providing a foundational resource for advancing wireless system design and RF-based perception, and catalyzing breakthroughs in RF-driven AI models.

Abstract

Current limitations in wireless modeling and radio frequency (RF)-based AI are primarily driven by a lack of high-quality, measurement-based datasets that connect RF signals to their physical environments. RF heatmaps, the typical form of such data, are high-dimensional and complex but lack the geometric and semantic context needed for interpretation, constraining the development of supervised machine learning models. To address this bottleneck, we propose a new class of multimodal datasets that combines RF measurements with auxiliary modalities like high-resolution cameras and lidar to bridge the gap between RF signals and their physical causes. The proposed data collection will span diverse indoor and outdoor environments, featuring both static and dynamic scenarios, including human activities ranging from walking to subtle gestures. By achieving precise spatial and temporal co-registration and creating digital replicas for voxel-level annotation, this dataset will enable transformative AI research. Key tasks include the forward problem of predicting RF heatmaps from visual data to revolutionize wireless system design, and the inverse problem of inferring scene semantics from RF signals, creating a new form of RF-based perception.

Problem

• RF interpretation
• multimodal dataset
• semantic annotation
• wireless modeling
• RF-based perception

Innovation

• multimodal dataset
• RF interpretation
• semantic annotation
• spatial-temporal co-registration
• RF-based perception