Learning to Label: A Reinforced Self-Evolving Framework for Semi-supervised Referring Expression Segmentation

2026-05-27

AI Summary

This work addresses the challenge of semi-supervised referring expression segmentation, where scarce annotations and unreliable pseudo-labels hinder precise alignment between language and pixels. To overcome this, the authors propose a reinforced self-evolution framework that, for the first time, formulates pseudo-label generation as a learnable reinforcement decision process. The approach leverages semantic-spatial priors and textual cues extracted by multimodal large language models to guide a hierarchical segmentation network. A reinforcement mechanism dynamically selects high-value pixel-level supervision signals, enabling joint optimization of the model and its pseudo-labels. Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg demonstrate substantial performance gains over existing methods, confirming the framework’s effectiveness and generalization capability.

Abstract

Semi-supervised referring expression segmentation (SS-RES) aims to achieve precise pixel-level language grounding under limited annotation, yet suffers from limited supervision and unreliable pseudo-labels when exploiting unlabeled image-text pairs. In this work, we propose Learning to Label, a reinforced self-evolving framework (L2L) that casts pseudo-label construction as a learnable decision-making process. To build foundational understanding, we leverage a multimodal large language model to extract semantic-spatial priors, which are instantiated as initial soft segmentation proposals and elevated, together with textual cues, into learnable guidance signals that condition a hierarchical segmentation network. To ensure stable learning, reinforced pseudo-label selection is formulated as an exploratory decision process that adaptively rewards high-utility pixel-level supervision based on multimodal priors and model predictions. This reinforced self-evolving loop enables joint optimization of the segmentation model and pseudo-labels, progressively enhancing label reliability under sparse supervision. Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg demonstrate improvements over existing methods, validating its effectiveness and generalization.

Problem

• semi-supervised learning
• referring expression segmentation
• pseudo-labeling
• pixel-level grounding
• limited annotation

Innovation

• semi-supervised referring expression segmentation
• reinforced self-evolving framework
• pseudo-label learning
• multimodal large language model
• hierarchical segmentation network