SketchSplat: 3D Edge Reconstruction via Differentiable Multi-view Sketch Splatting

2025-03-18

AI Summary

This work addresses the problem of parametric 3D edge reconstruction from calibrated multi-view images. We propose the first differentiable multi-view sketch splatting framework. Our method models 3D edges as differentiable parametric curves—parameterized by control points, scale, and opacity—and achieves end-to-end alignment between 2D edge maps and 3D geometry via Gaussian sampling and differentiable rasterization. To enhance efficiency and fidelity, we introduce an adaptive topology simplification operator that reduces sketch count while preserving structural accuracy, and design a highly robust 2D edge detector to improve input quality. Evaluated on CAD benchmarks, our approach comprehensively outperforms state-of-the-art methods: it achieves superior edge precision, completeness, and compactness, and produces 3D reconstructions that strictly align with the original multi-view 2D edge observations.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Learning on the Edge & Model Compression
• Planning, Routing, and Scheduling: Activity and Plan Recognition

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Edges are one of the most basic parametric primitives to describe structural information in 3D. In this paper, we study parametric 3D edge reconstruction from calibrated multi-view images. Previous methods usually reconstruct a 3D edge point set from multi-view 2D edge images, and then fit 3D edges to the point set. However, noise in the point set may cause gaps among fitted edges, and the recovered edges may not align with input multi-view images since the edge fitting depends only on the reconstructed 3D point set. To mitigate these problems, we propose SketchSplat, a method to reconstruct accurate, complete, and compact 3D edges via differentiable multi-view sketch splatting. We represent 3D edges as sketches, which are parametric lines and curves defined by attributes including control points, scales, and opacity. During edge reconstruction, we iteratively sample Gaussian points from a set of sketches and rasterize the Gaussians onto 2D edge images. Then the gradient of the image error with respect to the input 2D edge images can be back-propagated to optimize the sketch attributes. Our method bridges 2D edge images and 3D edges in a differentiable manner, which ensures that 3D edges align well with 2D images and leads to accurate and complete results. We also propose a series of adaptive topological operations and apply them along with the sketch optimization. The topological operations help reduce the number of sketches required while ensuring high accuracy, yielding a more compact reconstruction. Finally, we contribute an accurate 2D edge detector that improves the performance of both ours and existing methods. Experiments show that our method achieves state-of-the-art accuracy, completeness, and compactness on a benchmark CAD dataset.

Problem

• Reconstructs 3D edges from multi-view 2D images
• Addresses noise and alignment issues in edge fitting
• Ensures accurate, complete, and compact 3D edge reconstruction

Innovation

• Differentiable multi-view sketch splatting for 3D edges
• Adaptive topological operations for compact reconstruction
• Accurate 2D edge detector enhancing reconstruction performance