Video Inference for Human Mesh Recovery with Vision Transformer

2025-07-11

AI Summary

Human mesh recovery (HMR) is inherently challenging due to image ambiguity, and existing methods typically exploit either temporal information or kinematic priors in isolation, without jointly modeling both. This work proposes HMR-ViT, the first framework to unify temporal dynamics and human kinematic constraints for HMR. It extracts single-frame features via an image encoder to construct a spatio-temporal-kinematic feature map; introduces a channel-reordering matrix to explicitly group semantically similar motion patterns, thereby enhancing spatio-temporal-kinematic feature representation; and employs a Vision Transformer to encode this map and regress SMPL pose and shape parameters. Evaluated on 3DPW and Human3.6M, HMR-ViT achieves state-of-the-art or near-state-of-the-art performance, demonstrating that joint modeling of temporal and kinematic cues significantly improves both accuracy and generalizability in HMR.

Technology Category

• Computer Vision: Motion & Tracking
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Human Mesh Recovery (HMR) from an image is a challenging problem because of the inherent ambiguity of the task. Existing HMR methods utilized either temporal information or kinematic relationships to achieve higher accuracy, but there is no method using both. Hence, we propose "Video Inference for Human Mesh Recovery with Vision Transformer (HMR-ViT)" that can take into account both temporal and kinematic information. In HMR-ViT, a Temporal-kinematic Feature Image is constructed using feature vectors obtained from video frames by an image encoder. When generating the feature image, we use a Channel Rearranging Matrix (CRM) so that similar kinematic features could be located spatially close together. The feature image is then further encoded using Vision Transformer, and the SMPL pose and shape parameters are finally inferred using a regression network. Extensive evaluation on the 3DPW and Human3.6M datasets indicates that our method achieves a competitive performance in HMR.

Problem

• HMR lacks methods combining temporal and kinematic information
• Proposes HMR-ViT using Vision Transformer for both features
• Improves accuracy via Temporal-kinematic Feature Image and CRM

Innovation

• Uses Vision Transformer for human mesh recovery
• Integrates temporal and kinematic information jointly
• Employs Channel Rearranging Matrix for feature organization