MARBLE: Material Recomposition and Blending in CLIP-Space

2025-06-05

AI Summary

This work addresses fine-grained material editing from example images, enabling precise manipulation of object appearance and continuous control over physical material properties—including roughness, metallicness, transparency, and emissivity. Methodologically, we first identify material-attributive modules within a pre-trained diffusion UNet and introduce a novel CLIP-space material direction learning framework: a lightweight direction prediction network models semantic material directions in the CLIP embedding space, enabling module-level latent-space intervention in the UNet. This design supports multi-material blending, recombination, and cross-scene transfer in a single forward pass. Experiments demonstrate substantial improvements in material fidelity and controllability, outperforming state-of-the-art methods in both qualitative and quantitative evaluations—without requiring fine-tuning or additional training.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Manipulation

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Editing materials of objects in images based on exemplar images is an active area of research in computer vision and graphics. We propose MARBLE, a method for performing material blending and recomposing fine-grained material properties by finding material embeddings in CLIP-space and using that to control pre-trained text-to-image models. We improve exemplar-based material editing by finding a block in the denoising UNet responsible for material attribution. Given two material exemplar-images, we find directions in the CLIP-space for blending the materials. Further, we can achieve parametric control over fine-grained material attributes such as roughness, metallic, transparency, and glow using a shallow network to predict the direction for the desired material attribute change. We perform qualitative and quantitative analysis to demonstrate the efficacy of our proposed method. We also present the ability of our method to perform multiple edits in a single forward pass and applicability to painting. Project Page: https://marblecontrol.github.io/

Problem

• Editing object materials in images using exemplar images
• Blending materials via CLIP-space embeddings and text-to-image models
• Controlling fine-grained material attributes like roughness and transparency

Innovation

• Material blending in CLIP-space using embeddings
• Control pre-trained models via UNet material attribution
• Parametric control of material attributes with shallow network