DEARLi: Decoupled Enhancement of Recognition and Localization for Semi-supervised Panoptic Segmentation

2025-07-14

AI Summary

High-cost pixel-level annotations hinder the application of panoptic segmentation in few-shot settings. To address this, we propose a decoupled recognition-and-localization enhancement framework for semi-supervised panoptic segmentation: zero-shot category recognition is achieved via CLIP, while high-quality pseudo-labels are generated using SAM to warm-start a category-agnostic decoder; consistency training is further enforced via a mask Transformer. This work is the first to jointly integrate zero-shot classification and decoder warm-starting into semi-supervised panoptic segmentation. On ADE20K, our method achieves 29.9 PQ and 38.9 mIoU using only 158 annotated images—substantially outperforming prior approaches. Moreover, it reduces GPU memory consumption by 8×, demonstrating a novel paradigm where foundation models collaboratively enhance few-shot generalization.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Semi-Supervised Learning
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Pixel-level annotation is expensive and time-consuming. Semi-supervised segmentation methods address this challenge by learning models on few labeled images alongside a large corpus of unlabeled images. Although foundation models could further account for label scarcity, effective mechanisms for their exploitation remain underexplored. We address this by devising a novel semi-supervised panoptic approach fueled by two dedicated foundation models. We enhance recognition by complementing unsupervised mask-transformer consistency with zero-shot classification of CLIP features. We enhance localization by class-agnostic decoder warm-up with respect to SAM pseudo-labels. The resulting decoupled enhancement of recognition and localization (DEARLi) particularly excels in the most challenging semi-supervised scenarios with large taxonomies and limited labeled data. Moreover, DEARLi outperforms the state of the art in semi-supervised semantic segmentation by a large margin while requiring 8x less GPU memory, in spite of being trained only for the panoptic objective. We observe 29.9 PQ and 38.9 mIoU on ADE20K with only 158 labeled images. The source code is available at https://github.com/helen1c/DEARLi.

Problem

• Reducing reliance on expensive pixel-level annotations for segmentation
• Improving recognition and localization in semi-supervised panoptic segmentation
• Enhancing performance with limited labeled data and large taxonomies

Innovation

• Uses CLIP features for zero-shot classification
• Employs SAM pseudo-labels for decoder warm-up
• Decouples recognition and localization enhancement