Instance Segmentation of Scene Sketches Using Natural Image Priors

2025-02-13

AI Summary

This work addresses core challenges in scene sketch instance segmentation—namely, difficult pixel grouping, high sparsity, and significant stylistic variability—by proposing a class-agnostic sketch-domain fine-tuning framework coupled with depth-guided mask optimization. Methodologically, it introduces (i) a novel hierarchical ranking and occlusion-aware instance repair mechanism; (ii) the first synthetic sketch segmentation dataset encompassing diverse stroke styles, detail levels, and topological structures; and (iii) an integrated pipeline combining Mask R-CNN/YOLOv8 transfer learning, depth-map–guided mask refinement, and multi-level sketch topology modeling. Evaluated on the proposed dataset, the method achieves 62.3% mAP—substantially outperforming off-the-shelf general-image models. It further supports interactive editing operations including translation, scaling, and erasure. Both source code and the dataset are publicly released.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Multi-instance/Multi-view Learning
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Sketch segmentation involves grouping pixels within a sketch that belong to the same object or instance. It serves as a valuable tool for sketch editing tasks, such as moving, scaling, or removing specific components. While image segmentation models have demonstrated remarkable capabilities in recent years, sketches present unique challenges for these models due to their sparse nature and wide variation in styles. We introduce SketchSeg, a method for instance segmentation of raster scene sketches. Our approach adapts state-of-the-art image segmentation and object detection models to the sketch domain by employing class-agnostic fine-tuning and refining segmentation masks using depth cues. Furthermore, our method organizes sketches into sorted layers, where occluded instances are inpainted, enabling advanced sketch editing applications. As existing datasets in this domain lack variation in sketch styles, we construct a synthetic scene sketch segmentation dataset featuring sketches with diverse brush strokes and varying levels of detail. We use this dataset to demonstrate the robustness of our approach and will release it to promote further research in the field. Project webpage: https://sketchseg.github.io/sketch-seg/

Problem

• Instance segmentation of sparse sketches
• Adapting image models for sketch domain
• Creating a diverse sketch dataset

Innovation

• Adapts image segmentation models
• Uses depth cues for masks
• Creates synthetic sketch dataset