Differentiable Physics-Driven Human Representation for Millimeter-Wave Based Pose Estimation

2025-12-28

AI Summary

Millimeter-wave (mmWave) human pose estimation suffers from heatmap degradation due to multipath interference and hardware noise, while point-cloud-based representations yield sparse human feature encoding. Method: We propose Differentiable Physics-driven Human Representation (DIPR), modeling the human body as a collection of Gaussians jointly constrained by biomechanical kinematics and electromagnetic (EM) parameters. DIPR is the first differentiable representation integrating biomechanical priors with mmWave propagation physics, employing bidirectional optimization—kinematic initialization followed by physics-based rendering and closed-loop reconstruction—to suppress noise overfitting. Key techniques include Gaussian splatting modeling, differentiable EM simulation, and multi-objective kinematic optimization. Results: Evaluated on three benchmark datasets, DIPR serves as a plug-and-play module that consistently improves four state-of-the-art methods, significantly enhancing robustness against multipath effects and hardware noise.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: State Estimation

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

While millimeter-wave (mmWave) presents advantages for Human Pose Estimation (HPE) through its non-intrusive sensing capabilities, current mmWave-based HPE methods face limitations in two predominant input paradigms: Heatmap and Point Cloud (PC). Heatmap represents dense multi-dimensional features derived from mmWave, but is significantly affected by multipath propagation and hardware modulation noise. PC, a set of 3D points, is obtained by applying the Constant False Alarm Rate algorithm to the Heatmap, which suppresses noise but results in sparse human-related features. To address these limitations, we study the feasibility of providing an alternative input paradigm: Differentiable Physics-driven Human Representation (DIPR), which represents humans as an ensemble of Gaussian distributions with kinematic and electromagnetic parameters. Inspired by Gaussian Splatting, DIPR leverages human kinematic priors and mmWave propagation physics to enhance human features while mitigating non-human noise through two strategies: 1) We incorporate prior kinematic knowledge to initialize DIPR based on the Heatmap and establish multi-faceted optimization objectives, ensuring biomechanical validity and enhancing motion features. 2) We simulate complete mmWave processing pipelines, re-render a new Heatmap from DIPR, and compare it with the original Heatmap, avoiding spurious noise generation due to kinematic constraints overfitting. Experimental results on three datasets with four methods demonstrate that existing mmWave-based HPE methods can easily integrate DIPR and achieve superior performance.

Problem

• Proposes a differentiable physics-driven human representation for mmWave pose estimation
• Addresses limitations of heatmap and point cloud inputs in mmWave-based methods
• Enhances human features and reduces noise using kinematic and electromagnetic priors

Innovation

• Differentiable physics-driven Gaussian representation for humans
• Kinematic priors and electromagnetic parameters enhance features
• Simulated mmWave pipeline re-rendering mitigates noise overfitting