MatCLIP: Light- and Shape-Insensitive Assignment of PBR Material Models

2025-01-27

AI Summary

To address the susceptibility of automatic PBR material assignment for 3D models to illumination, viewpoint, and geometric variations, this paper proposes a shape- and illumination-invariant PBR material descriptor. Our method bridges the real-image domain and the physical material space via multi-view PBR rendering encoding, enabling end-to-end, part-agnostic material matching. Built upon an extended Alpha-CLIP architecture, it jointly leverages multi-condition PBR-rendered data and is trained using contrastive learning coupled with cross-domain feature alignment. On public benchmarks, our approach achieves a top-1 accuracy of 76.6%, outperforming state-of-the-art methods—including PhotoShape and MatAtlas—by over 15 percentage points. It generalizes effectively to mainstream 3D datasets such as ShapeNet and supports fully automatic, zero-prior material annotation.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Assigning realistic materials to 3D models remains a significant challenge in computer graphics. We propose MatCLIP, a novel method that extracts shape- and lighting-insensitive descriptors of Physically Based Rendering (PBR) materials to assign plausible textures to 3D objects based on images, such as the output of Latent Diffusion Models (LDMs) or photographs. Matching PBR materials to static images is challenging because the PBR representation captures the dynamic appearance of materials under varying viewing angles, shapes, and lighting conditions. By extending an Alpha-CLIP-based model on material renderings across diverse shapes and lighting, and encoding multiple viewing conditions for PBR materials, our approach generates descriptors that bridge the domains of PBR representations with photographs or renderings, including LDM outputs. This enables consistent material assignments without requiring explicit knowledge of material relationships between different parts of an object. MatCLIP achieves a top-1 classification accuracy of 76.6%, outperforming state-of-the-art methods such as PhotoShape and MatAtlas by over 15 percentage points on publicly available datasets. Our method can be used to construct material assignments for 3D shape datasets such as ShapeNet, 3DCoMPaT++, and Objaverse. All code and data will be released.

Problem

• PBR Materials
• 3D Modeling
• Lighting and Shape Invariance

Innovation

• MatCLIP
• PBR Material Modeling
• High Accuracy Material Mapping