MAEPose: Self-Supervised Spatiotemporal Learning for Human Pose Estimation on mmWave Video

2026-04-30

AI Summary

This work addresses the limitations of existing millimeter-wave radar-based human pose estimation methods, which rely on pre-extracted intermediate representations that discard spatiotemporal information from raw radar videos and are constrained by supervised learning paradigms. The authors propose MAEPose, the first approach to introduce masked autoencoding to millimeter-wave range-Doppler videos, enabling end-to-end self-supervised spatiotemporal modeling. MAEPose bypasses conventional intermediate signal processing and directly learns motion-aware generic representations from raw radar videos, coupled with a heatmap decoder for multi-frame pose estimation. Evaluated via leave-one-subject-out cross-validation across three datasets, MAEPose achieves a 22.1% improvement (p<0.05) over state-of-the-art methods in MPJPE and exhibits remarkable robustness, with only a 6.5% error increase under zero-shot bystander interference, demonstrating strong generalization capability.

Abstract

Millimetre-wave (mmWave) radar offers a more privacy-preserving alternative to RGB-based human pose estimation. However, existing methods typically rely on pre-extracted intermediate representations such as sparse point clouds or spectrogram images, where the rich spatiotemporal information naturally present in radar video streams is discarded for model learning, while such signal processing adds system complexity. In addition, existing solutions are mainly conducted in an end-to-end supervised manner without leveraging unlabelled raw video streams to learn generalized representations. In this study, we present MAEPose, a masked autoencoding-based human pose estimation approach that operates directly on mmWave spectrogram videos. MAEPose learns spatiotemporal motion-aware generalized representations from unlabelled radar video, and leverages its heatmap decoder for multi-frame pose estimation predictions. We evaluate it across three datasets based on leave-one-person-out cross-validation with rigorous statistical testing. MAEPose consistently outperforms state-of-the-art baselines by up to 22.1% in MPJPE p<0.05, and maintains robust accuracy under zero-shot bystander interference with only a 6.5% error increase. Ablation studies confirm that both the pre-training and the heatmap decoder contribute substantially, while modality analysis indicates that leveraging Range-Doppler video as input achieves better pose estimation performance than Range-Azimuth or their fusion, with lower computational cost.

Problem

• mmWave radar
• human pose estimation
• spatiotemporal learning
• self-supervised learning
• privacy-preserving

Innovation

• masked autoencoding
• self-supervised learning
• mmWave radar
• spatiotemporal representation
• human pose estimation