Refaçade: Editing Object with Given Reference Texture

2025-12-04

AI Summary

This paper introduces the novel task of *object retexing*, which aims to achieve precise and controllable transfer of local texture from a reference object to a target object, while preventing structural leakage and explicitly decoupling texture from geometry. Methodologically, we propose a texture-removal module trained via 3D mesh rendering, and introduce a jigsaw permutation strategy applied to reference images to disrupt global layout coherence—thereby enforcing focus on local texture patterns. We further integrate ControlNet to jointly enable geometry-appearance disentanglement and cross-domain texture transfer. Experiments demonstrate that our approach significantly outperforms strong baselines in both image and video editing, achieving superior visual fidelity, fine-grained controllability, and user-perceived quality. The method establishes a new paradigm for controllable texture editing, offering robust structural preservation and localized texture manipulation without geometric distortion.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Recent advances in diffusion models have brought remarkable progress in image and video editing, yet some tasks remain underexplored. In this paper, we introduce a new task, Object Retexture, which transfers local textures from a reference object to a target object in images or videos. To perform this task, a straightforward solution is to use ControlNet conditioned on the source structure and the reference texture. However, this approach suffers from limited controllability for two reasons: conditioning on the raw reference image introduces unwanted structural information, and it fails to disentangle the visual texture and structure information of the source. To address this problem, we propose Refaçade, a method that consists of two key designs to achieve precise and controllable texture transfer in both images and videos. First, we employ a texture remover trained on paired textured/untextured 3D mesh renderings to remove appearance information while preserving the geometry and motion of source videos. Second, we disrupt the reference global layout using a jigsaw permutation, encouraging the model to focus on local texture statistics rather than the global layout of the object. Extensive experiments demonstrate superior visual quality, precise editing, and controllability, outperforming strong baselines in both quantitative and human evaluations. Code is available at https://github.com/fishZe233/Refacade.

Problem

• Transfer local textures from reference to target objects
• Disentangle texture and structure information for precise editing
• Enhance controllability in image and video texture transfer

Innovation

• Texture remover preserves geometry, removes appearance from source
• Jigsaw permutation disrupts reference layout, focuses on local texture
• Method enables precise, controllable texture transfer in images and videos