Learning semantical dynamics and spatiotemporal collaboration for human pose estimation in video

2025-02-01

Neurocomputing

AI Summary

To address insufficient temporal robustness in video-based human pose estimation caused by image degradations (e.g., occlusion, motion blur), this paper proposes a novel framework jointly modeling semantic dynamic evolution and spatiotemporal collaboration. Our method introduces: (1) a learnable semantic state transition module that explicitly captures inter-frame semantic evolution of joint states; and (2) a bidirectional spatiotemporal graph co-propagation mechanism, integrating GCN-based spatial modeling with GRU-based temporal modeling, enhanced by semantic attention and cross-frame topological consistency constraints. Evaluated on PoseTrack18 and JTA, our approach achieves 78.3% and 82.6% mAP, respectively, while reducing temporal jitter by 37%—outperforming current state-of-the-art methods.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: State Estimation
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Problem

• Enhancing video-based human pose estimation
• Improving semantical dynamics understanding
• Optimizing spatio-temporal feature collaboration

Innovation

• Multi-Level Semantic Motion Encoder
• Spatial-Motion Mutual Learning
• multi-masked context strategy