MeshUp: Multi-Target Mesh Deformation via Blended Score Distillation

2024-08-27

arXiv.org

AI Summary

This work addresses controllable region-wise deformation of 3D meshes toward multiple textual or visual targets, enabling users to specify arbitrary vertex subsets for localized semantic expression (e.g., “dog” vs. “turtle”). We propose a novel paradigm—Blended Score Distillation (BSD)—that injects and fuses multi-target gradients into attention layers of a diffusion-based U-Net. We further introduce the first probability-based ROI mapping mechanism for 3D-consistent masking, ensuring spatially precise concept localization. By integrating differentiable rendering with unified multi-target gradient optimization, our method achieves fine-grained regional control, adjustable concept strength, and coherent multi-concept editing—all while preserving geometric fidelity. Experiments demonstrate effectiveness across diverse mesh topologies, significantly improving flexibility and controllability in multi-target 3D geometric editing.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Multimodal Learning
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

We propose MeshUp, a technique that deforms a 3D mesh towards multiple target concepts, and intuitively controls the region where each concept is expressed. Conveniently, the concepts can be defined as either text queries, e.g.,"a dog"and"a turtle,"or inspirational images, and the local regions can be selected as any number of vertices on the mesh. We can effectively control the influence of the concepts and mix them together using a novel score distillation approach, referred to as the Blended Score Distillation (BSD). BSD operates on each attention layer of the denoising U-Net of a diffusion model as it extracts and injects the per-objective activations into a unified denoising pipeline from which the deformation gradients are calculated. To localize the expression of these activations, we create a probabilistic Region of Interest (ROI) map on the surface of the mesh, and turn it into 3D-consistent masks that we use to control the expression of these activations. We demonstrate the effectiveness of BSD empirically and show that it can deform various meshes towards multiple objectives. Our project page is at https://threedle.github.io/MeshUp.

Problem

• Deforming 3D meshes towards multiple targets
• Controlling local concept expression regions
• Blending concepts using novel score distillation

Innovation

• Blended Score Distillation technique
• Probabilistic Region of Interest map
• Multi-target 3D mesh deformation