ProbRadarM3F: mmWave Radar based Human Skeletal Pose Estimation with Probability Map Guided Multi-Format Feature Fusion

2024-05-08

arXiv.org

AI Summary

Millimeter-wave (mmWave) radar-based human pose estimation has long suffered from low utilization of reflective signal information, resulting in a fundamental accuracy bottleneck. To address this, we propose a probabilistic graph-guided multi-format feature fusion framework that explicitly models implicit spatial priors embedded in radar signals—marking the first effort to exploit such priors. Specifically, we design a probabilistic graph-driven positional encoding module operating in parallel with conventional FFT-based time-frequency feature extraction. Subsequently, multi-source features are fused hierarchically via cascaded attention-weighted integration, enabling end-to-end regression of 14 anatomical keypoints. Evaluated on the HuPR dataset, our method achieves 69.9% Average Precision (AP), substantially outperforming existing radar-based approaches. This demonstrates that explicit modeling of spatial information is critical for advancing mmWave radar pose estimation accuracy.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Millimeter wave (mmWave) radar is a non-intrusive privacy and relatively convenient and inexpensive device, which has been demonstrated to be applicable in place of RGB cameras in human indoor pose estimation tasks. However, mmWave radar relies on the collection of reflected signals from the target, and the radar signals containing information is difficult to be fully applied. This has been a long-standing hindrance to the improvement of pose estimation accuracy. To address this major challenge, this paper introduces a probability map guided multi-format feature fusion model, ProbRadarM3F. This is a novel radar feature extraction framework using a traditional FFT method in parallel with a probability map based positional encoding method. ProbRadarM3F fuses the traditional heatmap features and the positional features, then effectively achieves the estimation of 14 keypoints of the human body. Experimental evaluation on the HuPR dataset proves the effectiveness of the model proposed in this paper, outperforming other methods experimented on this dataset with an AP of 69.9 %. The emphasis of our study is focusing on the position information that is not exploited before in radar singal. This provides direction to investigate other potential non-redundant information from mmWave rader.

Problem

• Improving human pose estimation accuracy using mmWave radar
• Addressing challenges in radar signal information utilization
• Fusing multi-format features for better keypoint detection

Innovation

• Probability map guided multi-format feature fusion
• Parallel FFT and positional encoding methods
• Fuses heatmap and positional features effectively