One-Shot Novel View and Pose Human Image Synthesis via 3D Prior Guided Diffusion Model

2026-06-17

AI Summary

This work addresses the challenges of low-quality synthesis and inaccurate reconstruction of complex poses and occluded regions in single-image human portrait generation under target poses and viewpoints. To this end, the authors propose a conditional denoising diffusion model that integrates 3D human priors. The method uniquely incorporates normal maps and color hints as geometric and appearance guidance signals within the diffusion generation process and introduces a self-reconstruction-based refinement strategy tailored to enhance fine details. Experimental results demonstrate that the proposed approach significantly outperforms existing state-of-the-art methods across multiple public benchmarks, exhibiting superior cross-dataset generalization and achieving high-fidelity synthesis of human images even under complex poses and severe occlusions.

Abstract

This paper addresses the challenge of one-shot novel view and pose human image synthesis. The existing methods transfer the reference human image to a target pose using a set of 2D pose keypoints or synthesize human images based on generalizable human NeRF which uses human model priors to extract point-wise features. However, pose transfer based methods can not handle complex human pose using ambiguous 2D pose as the condition, while generalizable human NeRFs may be inaccurate to recover occluded/invisiable human parts without extracted reliable features. To solve these problems, we propose a novel approach for novel view and pose synthesis from a singe human image via conditional denoising diffusion model. Our diffusion model divides the novel view and pose synthesis problem into a sequence of conditional denoising steps. Specifically, to generate humans with complex and arbitrary poses, we introduce 3D human priors, i.e., 3D normal map and color prompt, as geometry and color conditions into the generation process. By transferring the reference human into the target human with a series of diffusion steps, our diffusion model enables high-quality synthesis including the occluded/invisible parts. Further, we propose a self-reconstruction based customized refinement to enhance fine details when tested on novel persons.Experimental results on different public datasets demonstrate that our approach significantly outperforms previous methods and also shows better generalization ability across datasets. The code will be made publicly available at https://github.com/Yankeegsj/3DPGDM.

Problem

• one-shot
• novel view synthesis
• pose transfer
• human image synthesis
• occluded parts

Innovation

• 3D human prior
• diffusion model
• one-shot synthesis
• novel view synthesis
• pose transfer