Diff3DS: Generating View-Consistent 3D Sketch via Differentiable Curve Rendering

2024-05-24

arXiv.org

AI Summary

Existing 3D sketch generation methods suffer from heavy reliance on professional drafting expertise and geometric inconsistencies across multiple views. To address these challenges, this paper introduces the first end-to-end differentiable 3D sketch optimization framework. Our approach explicitly models 3D wireframes using rational Bézier curves in 3D space and integrates perspective projection with a custom differentiable rasterizer—enabling direct backpropagation of 2D image gradients to optimize 3D curve parameters. The framework supports both text- and image-conditioned cross-modal generation and is compatible with emerging supervision paradigms such as Score Distillation Sampling. Experiments demonstrate that our method generates high-fidelity, multi-view geometrically consistent 3D wireframe sketches for both text-to-3D-sketch and image-to-3D-sketch tasks, significantly outperforming prior approaches in visual quality and quantitative metrics.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Sampling/Simulation-based Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Responsible Web: Consent frameworks and practices on the web
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

3D sketches are widely used for visually representing the 3D shape and structure of objects or scenes. However, the creation of 3D sketch often requires users to possess professional artistic skills. Existing research efforts primarily focus on enhancing the ability of interactive sketch generation in 3D virtual systems. In this work, we propose Diff3DS, a novel differentiable rendering framework for generating view-consistent 3D sketch by optimizing 3D parametric curves under various supervisions. Specifically, we perform perspective projection to render the 3D rational B'ezier curves into 2D curves, which are subsequently converted to a 2D raster image via our customized differentiable rasterizer. Our framework bridges the domains of 3D sketch and raster image, achieving end-toend optimization of 3D sketch through gradients computed in the 2D image domain. Our Diff3DS can enable a series of novel 3D sketch generation tasks, including textto-3D sketch and image-to-3D sketch, supported by the popular distillation-based supervision, such as Score Distillation Sampling (SDS). Extensive experiments have yielded promising results and demonstrated the potential of our framework. Project page is at https://yiboz2001.github.io/Diff3DS/.

Problem

• Generates view-consistent 3D sketches without professional skills.
• Optimizes 3D parametric curves using differentiable rendering framework.
• Enables text-to-3D and image-to-3D sketch generation tasks.

Innovation

• Differentiable rendering for 3D sketch generation
• Optimizes 3D parametric curves under supervision
• Bridges 3D sketch and 2D raster image domains