mmHSense: Multi-Modal and Distributed mmWave ISAC Datasets for Human Sensing

2025-09-24

AI Summary

High-quality annotated data for human sensing in Integrated Sensing and Communication (ISAC) systems remains scarce, hindering progress in multimodal perception tasks. Method: This paper introduces and open-sources the first multimodal ISAC dataset tailored for four concurrent tasks—gesture recognition, person identification, pose estimation, and localization—collected via a distributed multimodal acquisition framework. It enables cross-scenario generalization and joint multi-task annotation for the first time in ISAC settings. Radar signal processing is leveraged to extract point clouds and Channel State Information (CSI) features; a parameter-efficient fine-tuning strategy is further proposed to facilitate rapid adaptation of multi-task models. Contribution/Results: Experiments demonstrate that the proposed approach achieves high accuracy while significantly reducing computational overhead, thereby enhancing the flexibility and scalability of ISAC systems in real-world deployments.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications

Abstract

This article presents mmHSense, a set of open labeled mmWave datasets to support human sensing research within Integrated Sensing and Communication (ISAC) systems. The datasets can be used to explore mmWave ISAC for various end applications such as gesture recognition, person identification, pose estimation, and localization. Moreover, the datasets can be used to develop and advance signal processing and deep learning research on mmWave ISAC. This article describes the testbed, experimental settings, and signal features for each dataset. Furthermore, the utility of the datasets is demonstrated through validation on a specific downstream task. In addition, we demonstrate the use of parameter-efficient fine-tuning to adapt ISAC models to different tasks, significantly reducing computational complexity while maintaining performance on prior tasks.

Problem

• Providing open mmWave datasets for human sensing research
• Enabling gesture recognition and pose estimation applications
• Developing efficient signal processing and deep learning methods

Innovation

• Multi-modal distributed mmWave ISAC datasets
• Parameter-efficient fine-tuning for model adaptation
• Reduced computational complexity with maintained performance