Weakly Supervised Segmentation as Semantic-Based Regularization

2026-05-13

AI Summary

This work addresses the challenges of low-quality pseudo-labels and difficulty in incorporating domain priors in weakly supervised semantic segmentation by proposing a neuro-symbolic approach that, for the first time, integrates differentiable fuzzy logic as a semantic regularization mechanism during foundation model fine-tuning. The method unifies diverse weak annotations—such as image-level tags and bounding boxes—with domain knowledge into continuous logical constraints, which are then used to refine pseudo-labels generated by the Segment Anything Model (SAM) and subsequently train a prompt-free segmentation model. Evaluated on Pascal VOC 2012 and REFUGE2, the approach substantially outperforms existing weakly supervised methods and even surpasses most fully supervised baselines, demonstrating its effectiveness and superiority in jointly modeling heterogeneous weak supervision signals and prior knowledge.

Abstract

Weakly supervised semantic segmentation (WSSS) trains dense pixel-level segmentation models from partial or coarse annotations such as bounding boxes, scribbles, or image-level tags. While recent work leverages foundation models such as the Segment Anything Model (SAM) to generate pseudo-labels, these approaches typically depend on heuristic prompt choices and offer limited ways to incorporate prior knowledge or heterogeneous labels. We address this gap by taking a neurosymbolic perspective: integrating differentiable fuzzy logic with deep segmentation models. Weak annotations and domain-specific priors are unified as continuous logical constraints that fine-tune SAM under weak supervision. The refined foundation model then produces improved pseudo-labels, from which we train a second-stage prompt-free segmentation model. Experiments on Pascal VOC 2012 and the REFUGE2 optic disc/cup segmentation dataset show that our logic-guided fine-tuning yields higher-quality pseudo-labels, leading to state-of-the-art segmentation accuracy that often exceeds densely supervised baselines.

Problem

• Weakly Supervised Semantic Segmentation
• Foundation Models
• Pseudo-labels
• Prior Knowledge
• Heterogeneous Labels

Innovation

• neurosymbolic learning
• weakly supervised segmentation
• differentiable fuzzy logic
• pseudo-label refinement
• foundation model fine-tuning