SAUGE: Taming SAM for Uncertainty-Aligned Multi-Granularity Edge Detection

2024-12-17

arXiv.org

AI Summary

Pixel-level label subjectivity in edge detection—arising from annotator preferences—and the challenge of modeling multi-granularity uncertainty hinder robust and generalizable edge prediction. Method: We first observe that SAM’s intermediate features inherently encode multi-granularity edge uncertainty; leveraging this, we propose an uncertainty-granularity explicit alignment mechanism. Specifically, we freeze the SAM backbone and introduce a lightweight adaptive feature fusion module to jointly regress multi-level features and normalize granularity-specific responses. Coupled with a linearly mixed pseudo-label construction strategy, our method enables arbitrary-granularity edge generation. Results: Our approach achieves significant improvements over state-of-the-art methods on BSDS500, Multicue, and NYUDv2, while demonstrating strong cross-dataset generalization. It simultaneously delivers high accuracy and robustness, effectively bridging subjective annotation variability and structural uncertainty across granularities.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Edge labels are typically at various granularity levels owing to the varying preferences of annotators, thus handling the subjectivity of per-pixel labels has been a focal point for edge detection. Previous methods often employ a simple voting strategy to diminish such label uncertainty or impose a strong assumption of labels with a pre-defined distribution, e.g., Gaussian. In this work, we unveil that the segment anything model (SAM) provides strong prior knowledge to model the uncertainty in edge labels. Our key insight is that the intermediate SAM features inherently correspond to object edges at various granularities, which reflects different edge options due to uncertainty. Therefore, we attempt to align uncertainty with granularity by regressing intermediate SAM features from different layers to object edges at multi-granularity levels. In doing so, the model can fully and explicitly explore diverse ``uncertainties'' in a data-driven fashion. Specifically, we inject a lightweight module (~ 1.5% additional parameters) into the frozen SAM to progressively fuse and adapt its intermediate features to estimate edges from coarse to fine. It is crucial to normalize the granularity level of human edge labels to match their innate uncertainty. For this, we simply perform linear blending to the real edge labels at hand to create pseudo labels with varying granularities. Consequently, our uncertainty-aligned edge detector can flexibly produce edges at any desired granularity (including an optimal one). Thanks to SAM, our model uniquely demonstrates strong generalizability for cross-dataset edge detection. Extensive experimental results on BSDS500, Muticue and NYUDv2 validate our model's superiority.

Problem

• Handles subjectivity in edge detection labels
• Aligns uncertainty with granularity using SAM features
• Produces edges at any desired granularity level

Innovation

• Leverages SAM for multi-granularity edge detection.
• Introduces lightweight module to adapt SAM features.
• Uses linear blending to align label granularity.