MeshPad: Interactive Sketch Conditioned Artistic-designed Mesh Generation and Editing

2025-03-03

AI Summary

This paper addresses the low editing efficiency and poor geometric consistency in sketch-based artistic 3D mesh creation, proposing MeshPad—a novel interactive system. Methodologically, it (1) introduces a vertex-aligned predictive decoding strategy enabling sub-second sketch-driven mesh editing; (2) pioneers a “delete–add” two-stage editing paradigm that preserves topological and geometric consistency while enhancing artistic controllability; and (3) employs a large-scale Transformer architecture leveraging triangle-mesh serialization and sketch-conditional modeling. Experiments demonstrate a >22% reduction in Chamfer distance compared to baselines; 90% of users prefer MeshPad in perceptual evaluations; and average single-step editing time is only a few seconds. Collectively, MeshPad significantly advances sketch-guided 3D mesh authoring by unifying speed, fidelity, and creative expressiveness.

Technology Category

• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

We introduce MeshPad, a generative approach that creates 3D meshes from sketch inputs. Building on recent advances in artistic-designed triangle mesh generation, our approach addresses the need for interactive artistic mesh creation. To this end, we focus on enabling consistent edits by decomposing editing into 'deletion' of regions of a mesh, followed by 'addition' of new mesh geometry. Both operations are invoked by simple user edits of a sketch image, facilitating an iterative content creation process and enabling the construction of complex 3D meshes. Our approach is based on a triangle sequence-based mesh representation, exploiting a large Transformer model for mesh triangle addition and deletion. In order to perform edits interactively, we introduce a vertex-aligned speculative prediction strategy on top of our additive mesh generator. This speculator predicts multiple output tokens corresponding to a vertex, thus significantly reducing the computational cost of inference and accelerating the editing process, making it possible to execute each editing step in only a few seconds. Comprehensive experiments demonstrate that MeshPad outperforms state-of-the-art sketch-conditioned mesh generation methods, achieving more than 22% mesh quality improvement in Chamfer distance, and being preferred by 90% of participants in perceptual evaluations.

Problem

• Generates 3D meshes from sketch inputs interactively
• Enables consistent mesh edits via deletion and addition
• Reduces computational cost with speculative prediction strategy

Innovation

• Generates 3D meshes from sketch inputs
• Uses Transformer model for mesh editing
• Implements vertex-aligned speculative prediction