Text2VDM: Text to Vector Displacement Maps for Expressive and Interactive 3D Sculpting

2025-02-27

AI Summary

Professional 3D sculpting relies on high-fidelity vector displacement map (VDM) brushes, yet existing generative models struggle to synthesize VDMs that are semantically accurate, geometrically faithful, and directly integrable into industrial pipelines. This paper introduces the first text-to-VDM brush generation framework. It employs differentiable planar mesh deformation for fine-grained geometric synthesis and pioneers the integration of classifier-free guidance (CFG) weighting into score distillation sampling (SDS) loss—effectively decoupling textual semantics from sub-object-level geometric structure and overcoming semantic coupling bottlenecks. The method enables high-fidelity, diverse VDM generation with plug-and-play outputs compatible with industry-standard software including Blender and ZBrush. Experiments demonstrate significant improvements over baselines in geometric detail fidelity, text–geometry alignment, and interactive practicality.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Web Mining and Content Analysis: Web data generation and simulation
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Professional 3D asset creation often requires diverse sculpting brushes to add surface details and geometric structures. Despite recent progress in 3D generation, producing reusable sculpting brushes compatible with artists' workflows remains an open and challenging problem. These sculpting brushes are typically represented as vector displacement maps (VDMs), which existing models cannot easily generate compared to natural images. This paper presents Text2VDM, a novel framework for text-to-VDM brush generation through the deformation of a dense planar mesh guided by score distillation sampling (SDS). The original SDS loss is designed for generating full objects and struggles with generating desirable sub-object structures from scratch in brush generation. We refer to this issue as semantic coupling, which we address by introducing classifier-free guidance (CFG) weighted blending of prompt tokens to SDS, resulting in a more accurate target distribution and semantic guidance. Experiments demonstrate that Text2VDM can generate diverse, high-quality VDM brushes for sculpting surface details and geometric structures. Our generated brushes can be seamlessly integrated into mainstream modeling software, enabling various applications such as mesh stylization and real-time interactive modeling.

Problem

• Generates vector displacement maps (VDMs) for 3D sculpting.
• Addresses semantic coupling in brush generation using CFG.
• Enables seamless integration of VDM brushes in modeling software.

Innovation

• Text-to-VDM brush generation
• Score Distillation Sampling
• Classifier-free guidance blending