Durian: Dual Reference-guided Portrait Animation with Attribute Transfer

2025-09-04

AI Summary

This work addresses the problem of zero-shot portrait animation—generating temporally coherent, high-fidelity facial attribute animations from a single reference image. Methodologically, we propose a dual-reference network that jointly models portrait structure and attribute features, integrated into the denoising process of diffusion models. To enhance generalization without triplet supervision, we introduce keypoint-guided mask expansion, spatial-appearance dual data augmentation, and a self-reconstruction training paradigm. Our approach is the first to enable simultaneous multi-attribute transfer in a single generation pass, overcoming the limitations of sequential, single-attribute editing. Experiments demonstrate strong generalization on in-the-wild images, significantly improved inter-frame consistency and occlusion robustness, and state-of-the-art performance on facial attribute transfer tasks.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We present Durian, the first method for generating portrait animation videos with facial attribute transfer from a given reference image to a target portrait in a zero-shot manner. To enable high-fidelity and spatially consistent attribute transfer across frames, we introduce dual reference networks that inject spatial features from both the portrait and attribute images into the denoising process of a diffusion model. We train the model using a self-reconstruction formulation, where two frames are sampled from the same portrait video: one is treated as the attribute reference and the other as the target portrait, and the remaining frames are reconstructed conditioned on these inputs and their corresponding masks. To support the transfer of attributes with varying spatial extent, we propose a mask expansion strategy using keypoint-conditioned image generation for training. In addition, we further augment the attribute and portrait images with spatial and appearance-level transformations to improve robustness to positional misalignment between them. These strategies allow the model to effectively generalize across diverse attributes and in-the-wild reference combinations, despite being trained without explicit triplet supervision. Durian achieves state-of-the-art performance on portrait animation with attribute transfer, and notably, its dual reference design enables multi-attribute composition in a single generation pass without additional training.

Problem

• Portrait animation with facial attribute transfer
• High-fidelity spatial consistency across frames
• Zero-shot generalization to diverse attribute combinations

Innovation

• Dual reference networks for spatial feature injection
• Mask expansion strategy using keypoint-conditioned generation
• Spatial and appearance-level transformations for robustness