Controllable Human-centric Keyframe Interpolation with Generative Prior

2025-06-03

AI Summary

Existing keyframe interpolation methods lack 3D geometric guidance, resulting in limited plausibility and controllability for complex human motion synthesis. This paper introduces PoseFuse3D-KI—the first diffusion-based keyframe interpolation framework integrated with 3D human priors. Its core innovations include: (i) a novel SMPL-X encoder that maps 3D human geometry into the 2D diffusion latent space; and (ii) a 3D-2D pose fusion network enabling human-centered, geometry-aware motion generation. The method jointly leverages video diffusion models, SMPL-X parameter encoding, a 3D-aware control network, and multimodal latent-space fusion. Evaluated on the newly constructed CHKI-Video dataset, PoseFuse3D-KI achieves a 9% PSNR improvement and a 38% LPIPS reduction over state-of-the-art methods. Ablation studies confirm that explicit 3D geometric guidance is critical for enhancing motion fidelity and temporal coherence.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Existing interpolation methods use pre-trained video diffusion priors to generate intermediate frames between sparsely sampled keyframes. In the absence of 3D geometric guidance, these methods struggle to produce plausible results for complex, articulated human motions and offer limited control over the synthesized dynamics. In this paper, we introduce PoseFuse3D Keyframe Interpolator (PoseFuse3D-KI), a novel framework that integrates 3D human guidance signals into the diffusion process for Controllable Human-centric Keyframe Interpolation (CHKI). To provide rich spatial and structural cues for interpolation, our PoseFuse3D, a 3D-informed control model, features a novel SMPL-X encoder that transforms 3D geometry and shape into the 2D latent conditioning space, alongside a fusion network that integrates these 3D cues with 2D pose embeddings. For evaluation, we build CHKI-Video, a new dataset annotated with both 2D poses and 3D SMPL-X parameters. We show that PoseFuse3D-KI consistently outperforms state-of-the-art baselines on CHKI-Video, achieving a 9% improvement in PSNR and a 38% reduction in LPIPS. Comprehensive ablations demonstrate that our PoseFuse3D model improves interpolation fidelity.

Problem

• Improving human motion interpolation with 3D guidance
• Enhancing control over synthesized human dynamics
• Integrating 3D geometry into 2D diffusion models

Innovation

• Integrates 3D human guidance into diffusion process
• Uses SMPL-X encoder for 3D to 2D transformation
• Combines 3D cues with 2D pose embeddings