DreamPose3D: Hallucinative Diffusion with Prompt Learning for 3D Human Pose Estimation

2025-11-12

AI Summary

To address the challenges of motion ambiguity and temporal discontinuity in 3D human pose estimation, this paper proposes an action-aware diffusion generative framework. Methodologically, it introduces motion-intent prompting and joint-dynamics-aware attention to capture fine-grained motion semantics from ambiguous 2D inputs; integrates a “hallucination-based” temporal decoding strategy for high-fidelity 3D sequence generation; and jointly optimizes joint affinity modeling and temporal consistency constraints in an end-to-end manner. The framework achieves state-of-the-art performance on Human3.6M and MPI-3DHP benchmarks. Notably, it demonstrates superior robustness under severe noise and complex motions—e.g., broadcast-level baseball videos—while significantly improving cross-scenario generalization and spatiotemporal coherence.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Computer Vision: Diffusion Models for Vision
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Accurate 3D human pose estimation remains a critical yet unresolved challenge, requiring both temporal coherence across frames and fine-grained modeling of joint relationships. However, most existing methods rely solely on geometric cues and predict each 3D pose independently, which limits their ability to resolve ambiguous motions and generalize to real-world scenarios. Inspired by how humans understand and anticipate motion, we introduce DreamPose3D, a diffusion-based framework that combines action-aware reasoning with temporal imagination for 3D pose estimation. DreamPose3D dynamically conditions the denoising process using task-relevant action prompts extracted from 2D pose sequences, capturing high-level intent. To model the structural relationships between joints effectively, we introduce a representation encoder that incorporates kinematic joint affinity into the attention mechanism. Finally, a hallucinative pose decoder predicts temporally coherent 3D pose sequences during training, simulating how humans mentally reconstruct motion trajectories to resolve ambiguity in perception. Extensive experiments on benchmarked Human3.6M and MPI-3DHP datasets demonstrate state-of-the-art performance across all metrics. To further validate DreamPose3D's robustness, we tested it on a broadcast baseball dataset, where it demonstrated strong performance despite ambiguous and noisy 2D inputs, effectively handling temporal consistency and intent-driven motion variations.

Problem

• Estimating 3D human poses with temporal coherence and joint relationship modeling
• Resolving ambiguous motions and generalizing to real-world scenarios
• Handling temporal consistency and intent-driven motion variations effectively

Innovation

• Diffusion-based framework with action-aware reasoning
• Kinematic joint affinity in attention mechanism
• Hallucinative pose decoder for temporal coherence