MITO: Enabling Non-Line-of-Sight Perception using Millimeter-waves through Real-World Datasets and Simulation Tools

2025-02-14

AI Summary

This work addresses critical bottlenecks in millimeter-wave (mmWave) non-line-of-sight (NLoS) sensing research—namely, scarcity of real-world data and challenges in cross-disciplinary physical modeling. We introduce the first real-world mmWave image dataset featuring fine-grained 3D annotations across multiple frequency bands (580+ samples) and release an open-source, physics-informed mmWave simulation engine with high fidelity (F-score: 94%). To bridge the synthetic-to-real domain gap, we propose a triangle-mesh-based synthetic data generation method and a SAM-adapted mmWave segmentation framework—establishing the first viable synthetic-training-to-real-testing pipeline for NLoS perception. On challenging occluded scenarios (e.g., cardboard, fabric), our method achieves 92.6% mIoU and 64% recall for mmWave image segmentation, and 85% accuracy for NLoS object recognition. This work provides a foundational data resource and a reproducible technical paradigm for mmWave vision research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Segmentation
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Web Mining and Content Analysis: Web data generation and simulation
• Security and Privacy: Data transparency and provenance

Abstract

We present MITO, the first dataset of multi-spectral millimeter-wave (mmWave) images of everyday objects. Unlike visible light, mmWave signals can image through everyday occlusions (e.g., cardboard boxes, fabric, plastic). However, due to the dearth of publicly-available mmWave images and the interdisciplinary challenges in collecting and processing mmWave signals, it remains difficult today for computer vision researchers to develop mmWave-based non-line-of-sight perception algorithms and models. To overcome these challenges, we introduce a real-world dataset and open-source simulation tool for mmWave imaging. The dataset is acquired using a UR5 robotic arm with two mmWave radars operating at different frequencies and an RGB-D camera. Through a signal processing pipeline, we capture and create over 580 real-world 3D mmWave images from over 76 different objects in the YCB dataset, a standard dataset for robotics manipulation. We provide real-world mmWave images in line-of-sight and non-line-of-sight, as well as RGB-D images and ground truth segmentation masks. We also develop an open-source simulation tool that can be used to generate synthetic mmWave images for any 3D triangle mesh, which achieves a median F-Score of 94% when compared to real-world mmWave images. We show the usefulness of this dataset and simulation tool in multiple CV tasks in non-line-of-sight. First, we perform object segmentation for mmWave images using the segment anything model (SAM), and achieve a median precision and recall of 92.6% and 64%. Second, we train a classifier that can recognize objects in non-line-of-sight. It is trained on synthetic images and can classify real-world images with 85% accuracy. We believe MITO will be a valuable resource for computer vision researchers in developing non-line-of-sight perception, similar to how early camera-based datasets shaped the field.

Problem

• Develop non-line-of-sight perception algorithms
• Overcome lack of mmWave image datasets
• Create simulation tool for mmWave imaging

Innovation

• Multi-spectral mmWave imaging dataset
• Open-source mmWave simulation tool
• Non-line-of-sight object recognition