3DSwapping: Texture Swapping For 3D Object From Single Reference Image

2025-03-24

AI Summary

This paper addresses two key challenges in single-reference-image-driven 3D texture swapping: cross-view texture inconsistency and low fidelity to the reference texture. To this end, we propose a diffusion-based progressive 3D texture editing framework. Methodologically, it introduces a novel triple-cooperative mechanism: (1) progressive implicit 3D generation for coarse-to-fine texture reconstruction; (2) feature-difference-driven view-consistency gradient guidance to explicitly enforce multi-view texture coherence; and (3) prompt-tuning gradient guidance with learnable text tokens to enhance preservation of fine-grained texture features from the reference image. Extensive qualitative and quantitative evaluations on multiple benchmarks demonstrate that our approach significantly improves texture fidelity (−12.7% LPIPS) and structural consistency (−9.3% Chamfer Distance), achieving, for the first time, high-fidelity and cross-view-consistent single-image-driven 3D texture swapping.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

3D texture swapping allows for the customization of 3D object textures, enabling efficient and versatile visual transformations in 3D editing. While no dedicated method exists, adapted 2D editing and text-driven 3D editing approaches can serve this purpose. However, 2D editing requires frame-by-frame manipulation, causing inconsistencies across views, while text-driven 3D editing struggles to preserve texture characteristics from reference images. To tackle these challenges, we introduce 3DSwapping, a 3D texture swapping method that integrates: 1) progressive generation, 2) view-consistency gradient guidance, and 3) prompt-tuned gradient guidance. To ensure view consistency, our progressive generation process starts by editing a single reference image and gradually propagates the edits to adjacent views. Our view-consistency gradient guidance further reinforces consistency by conditioning the generation model on feature differences between consistent and inconsistent outputs. To preserve texture characteristics, we introduce prompt-tuning-based gradient guidance, which learns a token that precisely captures the difference between the reference image and the 3D object. This token then guides the editing process, ensuring more consistent texture preservation across views. Overall, 3DSwapping integrates these novel strategies to achieve higher-fidelity texture transfer while preserving structural coherence across multiple viewpoints. Extensive qualitative and quantitative evaluations confirm that our three novel components enable convincing and effective 2D texture swapping for 3D objects. Code will be available upon acceptance.

Problem

• Enables 3D texture swapping from single reference image
• Resolves view inconsistencies in 2D frame-by-frame editing
• Preserves texture characteristics in text-driven 3D editing

Innovation

• Progressive generation for view consistency
• View-consistency gradient guidance
• Prompt-tuned gradient guidance for textures