GRPose: Learning Graph Relations for Human Image Generation with Pose Priors

2024-08-29

arXiv.org

AI Summary

To address inconsistent pose alignment and the trade-off between visual quality and geometric accuracy in controllable human image generation with diffusion models, this work pioneers modeling pose priors as graph structures and establishing a topological mapping between such graphs and the diffusion latent space. We propose a Progressive Graph Integrator (PGI) to capture multi-scale pose relationships and introduce a pose-aware loss for end-to-end optimization of pose fidelity. Our method integrates graph neural networks, Adapter-based fine-tuning, a pre-trained pose estimator, and a custom graph propagation mechanism. Evaluated on Human-Art and LAION-Human, our approach significantly outperforms state-of-the-art methods, achieving consistent improvements across quantitative metrics—including FID and keypoint error (KP-Error)—while qualitative results demonstrate simultaneous enhancement of pose precision and visual naturalness. The code is publicly available.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Recent methods using diffusion models have made significant progress in human image generation with various control signals such as pose priors. However, existing efforts are still struggling to generate high-quality images with consistent pose alignment, resulting in unsatisfactory output. In this paper, we propose a framework that delves into the graph relations of pose priors to provide control information for human image generation. The main idea is to establish a graph topological structure between the pose priors and latent representation of diffusion models to capture the intrinsic associations between different pose parts. A Progressive Graph Integrator (PGI) is designed to learn the spatial relationships of the pose priors with the graph structure, adopting a hierarchical strategy within an Adapter to gradually propagate information across different pose parts. Besides, a pose perception loss is introduced based on a pretrained pose estimation network to minimize the pose differences. Extensive qualitative and quantitative experiments conducted on the Human-Art and LAION-Human datasets clearly demonstrate that our model can achieve significant performance improvement over the latest benchmark models. The code is available at url{https://xiangchenyin.github.io/GRPose/}.

Problem

• Diffusion Models
• High-quality Human Image Generation
• Pose Accuracy

Innovation

• GRPose
• Progressive Graph Integrator
• Pose-Accurate Image Synthesis