LangPose: Language-Aligned Motion for Robust 3D Human Pose Estimation

2024-08-31

AI Summary

2D-to-3D human pose estimation is ill-posed due to depth ambiguity and occlusion, and existing methods—relying heavily on spatiotemporal consistency—exhibit limited robustness under severe occlusion or high-motion dynamics. This paper proposes a language-action-pose joint alignment framework that, for the first time, aligns motion sequence embeddings with fine-grained action-text embeddings to inject semantic priors for disambiguation. We introduce a masked body-part and temporal-window modeling mechanism that activates semantic compensation when spatiotemporal constraints fail. Our approach employs multi-stage training (action recognition + pose pretraining → unsupervised 3D fine-tuning), cross-modal contrastive alignment, motion sequence masking, and joint reconstruction-contrastive optimization. On Human3.6M with detected 2D inputs, we achieve 36.7 mm MPJPE; on MPI-INF-3DHP with ground-truth 2D inputs, we attain 15.5 mm MPJPE—setting new state-of-the-art performance.

Technology Category

• Computer Vision: 3D Computer Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

2D-to-3D human pose lifting is an ill-posed problem due to depth ambiguity and occlusion. Existing methods relying on spatial and temporal consistency alone are insufficient to resolve these problems especially in the presence of significant occlusions or high dynamic actions. Semantic information, however, offers a complementary signal that can help disambiguate such cases. To this end, we propose LangPose, a framework that leverages action knowledge by aligning motion embeddings with text embeddings of fine-grained action labels. LangPose operates in two stages: pretraining and fine-tuning. In the pretraining stage, the model simultaneously learns to recognize actions and reconstruct 3D poses from masked and noisy 2D poses. During the fine-tuning stage, the model is further refined using real-world 3D human pose estimation datasets without action labels. Additionally, our framework incorporates masked body parts and masked time windows in motion modeling, encouraging the model to leverage semantic information when spatial and temporal consistency is unreliable. Experiments demonstrate the effectiveness of LangPose, achieving SOTA level performance in 3D pose estimation on public datasets, including Human3.6M and MPI-INF-3DHP. Specifically, LangPose achieves an MPJPE of 36.7mm on Human3.6M with detected 2D poses as input and 15.5mm on MPI-INF-3DHP with ground-truth 2D poses as input.

Problem

• Resolving depth ambiguity in 2D-to-3D pose estimation using semantic alignment
• Addressing occlusion challenges by integrating fine-grained action knowledge
• Improving robustness for high-dynamic actions through language-motion embedding

Innovation

• Aligns motion embeddings with text action labels
• Uses pretraining with masked poses and action recognition
• Incorporates masked body parts and time windows