CasTex: Cascaded Text-to-Texture Synthesis via Explicit Texture Maps and Physically-Based Shading

2025-04-09

AI Summary

This work addresses the lack of physical plausibility and severe visual artifacts in text-to-texture synthesis. Existing approaches—e.g., score distillation sampling—rely on implicit texture parameterizations, which inherently introduce artifacts and necessitate strong regularization. To overcome these limitations, we propose CasTex, a cascaded diffusion model that abandons implicit modeling in favor of explicit texture map representation. CasTex integrates differentiable rasterization with a physically based rendering (PBR) pipeline for end-to-end optimization. Its cascaded architecture first generates a coarse base texture and then refines it with high-frequency details, balancing global coherence and local fidelity. On public benchmarks, CasTex significantly outperforms prior methods: it eliminates artifacts without auxiliary regularization and produces textures exhibiting lighting consistency, geometry awareness, and high-detail fidelity. This establishes a new paradigm for text-driven, physically grounded texture synthesis.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

This work investigates text-to-texture synthesis using diffusion models to generate physically-based texture maps. We aim to achieve realistic model appearances under varying lighting conditions. A prominent solution for the task is score distillation sampling. It allows recovering a complex texture using gradient guidance given a differentiable rasterization and shading pipeline. However, in practice, the aforementioned solution in conjunction with the widespread latent diffusion models produces severe visual artifacts and requires additional regularization such as implicit texture parameterization. As a more direct alternative, we propose an approach using cascaded diffusion models for texture synthesis (CasTex). In our setup, score distillation sampling yields high-quality textures out-of-the box. In particular, we were able to omit implicit texture parameterization in favor of an explicit parameterization to improve the procedure. In the experiments, we show that our approach significantly outperforms state-of-the-art optimization-based solutions on public texture synthesis benchmarks.

Problem

• Generating physically-based texture maps using diffusion models
• Achieving realistic model appearances under varying lighting conditions
• Overcoming visual artifacts in text-to-texture synthesis

Innovation

• Cascaded diffusion models for texture synthesis
• Explicit texture parameterization improves quality
• Score distillation sampling without implicit regularization