Poseidon: A ViT-based Architecture for Multi-Frame Pose Estimation with Adaptive Frame Weighting and Multi-Scale Feature Fusion

2025-01-14

AI Summary

To address insufficient temporal modeling in video-based human pose estimation—which compromises accuracy and stability—this paper proposes a temporally enhanced Transformer framework extending ViTPose. The method introduces three key innovations: (1) an adaptive frame weighting mechanism that dynamically selects salient frames to mitigate motion blur and occlusion; (2) a multi-scale feature fusion module that jointly models fine-grained joint localization and high-level semantic context; and (3) a cross-frame cross-attention module that explicitly captures long-range temporal dependencies and enforces pose consistency across frames. Evaluated on PoseTrack18 and PoseTrack21, the framework achieves 87.8 and 88.3 mAP, respectively—setting new state-of-the-art results at the time of publication. These results validate the effectiveness and robustness of the proposed temporal modeling paradigm for video pose estimation.

Technology Category

• Computer Vision: Motion & Tracking
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Planning, Routing, and Scheduling: Temporal Planning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Human pose estimation, a vital task in computer vision, involves detecting and localising human joints in images and videos. While single-frame pose estimation has seen significant progress, it often fails to capture the temporal dynamics for understanding complex, continuous movements. We propose Poseidon, a novel multi-frame pose estimation architecture that extends the ViTPose model by integrating temporal information for enhanced accuracy and robustness to address these limitations. Poseidon introduces key innovations: (1) an Adaptive Frame Weighting (AFW) mechanism that dynamically prioritises frames based on their relevance, ensuring that the model focuses on the most informative data; (2) a Multi-Scale Feature Fusion (MSFF) module that aggregates features from different backbone layers to capture both fine-grained details and high-level semantics; and (3) a Cross-Attention module for effective information exchange between central and contextual frames, enhancing the model's temporal coherence. The proposed architecture improves performance in complex video scenarios and offers scalability and computational efficiency suitable for real-world applications. Our approach achieves state-of-the-art performance on the PoseTrack21 and PoseTrack18 datasets, achieving mAP scores of 88.3 and 87.8, respectively, outperforming existing methods.

Problem

• Human Pose Estimation
• Complex Continuous Actions
• Temporal Variability

Innovation

• Auto Weight Adjustment
• Multi-Scale Feature Fusion
• Contextual Information Interaction