GuideFlow3D: Optimization-Guided Rectified Flow For Appearance Transfer

2025-10-17

AI Summary

Large geometric discrepancies in cross-modal appearance transfer cause severe texture and detail distortion. Method: We propose a training-free, general guidance sampling intervention framework leveraging pre-trained image/text-conditional diffusion models for geometry-robust texture transfer. Our core innovation is a differentiable geometry-appearance alignment loss that serves as a corrective flow guidance signal, integrating part-aware and self-similarity priors with a periodic gradient optimization mechanism. Contribution/Results: The method achieves state-of-the-art performance in both qualitative and quantitative evaluations, accurately recovering fine-grained textures and structural details. To enable scalable, objective assessment, we develop a GPT-driven automated evaluation system, validated via user studies to strongly align with human preferences (Spearman’s ρ = 0.92).

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

Transferring appearance to 3D assets using different representations of the appearance object - such as images or text - has garnered interest due to its wide range of applications in industries like gaming, augmented reality, and digital content creation. However, state-of-the-art methods still fail when the geometry between the input and appearance objects is significantly different. A straightforward approach is to directly apply a 3D generative model, but we show that this ultimately fails to produce appealing results. Instead, we propose a principled approach inspired by universal guidance. Given a pretrained rectified flow model conditioned on image or text, our training-free method interacts with the sampling process by periodically adding guidance. This guidance can be modeled as a differentiable loss function, and we experiment with two different types of guidance including part-aware losses for appearance and self-similarity. Our experiments show that our approach successfully transfers texture and geometric details to the input 3D asset, outperforming baselines both qualitatively and quantitatively. We also show that traditional metrics are not suitable for evaluating the task due to their inability of focusing on local details and comparing dissimilar inputs, in absence of ground truth data. We thus evaluate appearance transfer quality with a GPT-based system objectively ranking outputs, ensuring robust and human-like assessment, as further confirmed by our user study. Beyond showcased scenarios, our method is general and could be extended to different types of diffusion models and guidance functions.

Problem

• Transferring appearance to 3D assets with geometric differences
• Improving texture and detail transfer using optimization guidance
• Evaluating appearance transfer without ground truth data

Innovation

• Optimization-guided rectified flow for appearance transfer
• Training-free guidance during sampling process
• Part-aware and self-similarity loss functions