S3D: Sketch-Driven 3D Model Generation

2025-05-07

AI Summary

This paper addresses the significant ambiguity in 3D reconstruction from 2D hand-drawn sketches, arising from their inherent ambiguity and sparsity. To tackle this challenge, we propose an end-to-end sketch-to-3D generation framework. Methodologically: (1) we introduce a style-alignment loss that explicitly bridges sketch encoding features with the initial implicit/explicit representations in the 3D generator; (2) we adopt a U-Net architecture conditioned on patch-wise segmentation mask prediction, augmented with multi-strategy sketch data augmentation to enhance generalization; (3) we enforce cross-domain feature alignment and joint optimization of sketch and 3D representations. Extensive experiments on ShapeNet and Sketch2CAD benchmarks demonstrate substantial improvements in reconstruction accuracy and multi-view visual consistency, achieving state-of-the-art performance. The source code is publicly available.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: 3D Computer Vision
• Natural Language Processing: Generation

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Generating high-quality 3D models from 2D sketches is a challenging task due to the inherent ambiguity and sparsity of sketch data. In this paper, we present S3D, a novel framework that converts simple hand-drawn sketches into detailed 3D models. Our method utilizes a U-Net-based encoder-decoder architecture to convert sketches into face segmentation masks, which are then used to generate a 3D representation that can be rendered from novel views. To ensure robust consistency between the sketch domain and the 3D output, we introduce a novel style-alignment loss that aligns the U-Net bottleneck features with the initial encoder outputs of the 3D generation module, significantly enhancing reconstruction fidelity. To further enhance the network's robustness, we apply augmentation techniques to the sketch dataset. This streamlined framework demonstrates the effectiveness of S3D in generating high-quality 3D models from sketch inputs. The source code for this project is publicly available at https://github.com/hailsong/S3D.

Problem

• Convert 2D sketches into detailed 3D models
• Address ambiguity and sparsity in sketch data
• Enhance reconstruction fidelity with style-alignment loss

Innovation

• U-Net-based encoder-decoder for sketch-to-mask conversion
• Style-alignment loss enhances 3D reconstruction fidelity
• Augmentation techniques improve sketch dataset robustness