mmWEAVER: Environment-Specific mmWave Signal Synthesis from a Photo and Activity Description

2025-12-09

AI Summary

Millimeter-wave (mmWave) radar signal acquisition is challenging, physical simulation is computationally expensive, and modeling complex-valued radar signals—characterized by high dimensionality, sparsity, and phase sensitivity—remains difficult. To address these issues, we propose a semantic-geometric dual-prior-driven implicit neural representation (INR) framework. Given RGB-D images and textual descriptions as input, our method employs a dynamic hypernetwork to jointly encode geometric structure and semantic motion priors (leveraging MotionGPT), enabling direct synthesis of environment-specific, high-fidelity complex-valued mmWave radar signals. To our knowledge, this is the first approach to achieve cross-resolution phase consistency preservation and 49× parameter compression. Experiments show a complex SSIM of 0.88 and PSNR of 35 dB; downstream tasks yield a 7% improvement in activity recognition accuracy and a 15% reduction in pose estimation error. Inference is 6–35× faster than conventional physics-based simulation.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Web data generation and simulation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Realistic signal generation and dataset augmentation are essential for advancing mmWave radar applications such as activity recognition and pose estimation, which rely heavily on diverse, and environment-specific signal datasets. However, mmWave signals are inherently complex, sparse, and high-dimensional, making physical simulation computationally expensive. This paper presents mmWeaver, a novel framework that synthesizes realistic, environment-specific complex mmWave signals by modeling them as continuous functions using Implicit Neural Representations (INRs), achieving up to 49-fold compression. mmWeaver incorporates hypernetworks that dynamically generate INR parameters based on environmental context (extracted from RGB-D images) and human motion features (derived from text-to-pose generation via MotionGPT), enabling efficient and adaptive signal synthesis. By conditioning on these semantic and geometric priors, mmWeaver generates diverse I/Q signals at multiple resolutions, preserving phase information critical for downstream tasks such as point cloud estimation and activity classification. Extensive experiments show that mmWeaver achieves a complex SSIM of 0.88 and a PSNR of 35 dB, outperforming existing methods in signal realism while improving activity recognition accuracy by up to 7% and reducing human pose estimation error by up to 15%, all while operating 6-35 times faster than simulation-based approaches.

Problem

• Synthesizes realistic mmWave signals from photos and activity descriptions
• Compresses mmWave signal representation using Implicit Neural Representations
• Enables efficient dataset augmentation for radar-based applications

Innovation

• Uses Implicit Neural Representations for signal compression
• Employs hypernetworks with RGB-D and text-to-pose inputs
• Generates multi-resolution I/Q signals preserving phase information