AnchorCrafter: Animate CyberAnchors Saling Your Products via Human-Object Interacting Video Generation

2024-11-26

arXiv.org

AI Summary

To address object appearance distortion and unnatural human-object interaction in pose-guided person video generation for e-commerce applications, this paper proposes the first video generation framework explicitly designed for human-object interaction (HOI) modeling. Methodologically, we introduce an HOI-appearance-aware representation learning scheme and an HOI-motion injection mechanism, coupled with an HOI-region reweighting loss to enable multi-view object disentanglement, complex interaction trajectory modeling, and robust handling of mutual occlusion. Our diffusion-based 2D video generation architecture integrates multi-view appearance encoding, motion-conditioned feature injection, and region-aware adversarial training. Experiments demonstrate that our approach significantly outperforms state-of-the-art methods on object shape/appearance fidelity and person identity consistency metrics. The generated videos exhibit realistic human-object interactions and commercially viable visual quality.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Humans and AI: Interaction Techniques and Devices
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

The automatic generation of anchor-style product promotion videos presents promising opportunities in online commerce, advertising, and consumer engagement. However, this remains a challenging task despite significant advancements in pose-guided human video generation. In addressing this challenge, we identify the integration of human-object interactions (HOI) into pose-guided human video generation as a core issue. To this end, we introduce AnchorCrafter, a novel diffusion-based system designed to generate 2D videos featuring a target human and a customized object, achieving high visual fidelity and controllable interactions. Specifically, we propose two key innovations: the HOI-appearance perception, which enhances object appearance recognition from arbitrary multi-view perspectives and disentangles object and human appearance, and the HOI-motion injection, which enables complex human-object interactions by overcoming challenges in object trajectory conditioning and inter-occlusion management. Additionally, we introduce the HOI-region reweighting loss, a training objective that enhances the learning of object details. Extensive experiments demonstrate that our proposed system outperforms existing methods in preserving object appearance and shape awareness, while simultaneously maintaining consistency in human appearance and motion. Project page: https://cangcz.github.io/Anchor-Crafter/

Problem

• Integrate human-object interactions into pose-guided video generation
• Enhance object appearance recognition from multi-view perspectives
• Enable complex human-object interactions with trajectory conditioning

Innovation

• Diffusion-based 2D video generation system
• HOI-appearance perception for multi-view recognition
• HOI-motion injection for complex interactions