TALENT: Target-aware Efficient Tuning for Referring Image Segmentation

2026-04-01

AI Summary

This work addresses the “non-target activation” (NTA) problem arising from parameter-efficient fine-tuning in referring expression segmentation. The proposed TALENT framework formally defines and names the NTA phenomenon for the first time, introducing a target-aware efficient fine-tuning mechanism that synergistically integrates a rectified cost aggregator (RCA), context-aware pairwise consistency learning, and target-centric contrastive learning to jointly optimize text-guided object localization and semantic alignment. By constructing a text-guided affinity graph, TALENT effectively suppresses activations from irrelevant objects, significantly enhancing instance-level segmentation accuracy. On the G-Ref validation set, the method achieves a 2.5% improvement in mIoU and outperforms existing approaches across multiple metrics, demonstrating the efficacy of the proposed mechanism.

Abstract

Referring image segmentation aims to segment specific targets based on a natural text expression. Recently, parameter-efficient tuning (PET) has emerged as a promising paradigm. However, existing PET-based methods often suffer from the fact that visual features can't emphasize the text-referred target instance but activate co-category yet unrelated objects. We analyze and quantify this problem, terming it the `non-target activation' (NTA) issue. To address this, we propose a novel framework, TALENT, which utilizes target-aware efficient tuning for PET-based RIS. Specifically, we first propose a Rectified Cost Aggregator (RCA) to efficiently aggregate text-referred features. Then, to calibrate `NTA' into accurate target activation, we adopt a Target-aware Learning Mechanism (TLM), including contextual pairwise consistency learning and target-centric contrastive learning. The former uses the sentence-level text feature to achieve a holistic understanding of the referent and constructs a text-referred affinity map to optimize the semantic association of visual features. The latter further enhances target localization to discover the distinct instance while suppressing associations with other unrelated ones. The two objectives work in concert and address `NTA' effectively. Extensive evaluations show that TALENT outperforms existing methods across various metrics (e.g., 2.5\% mIoU gains on G-Ref val set). Our codes will be released at: https://github.com/Kimsure/TALENT.

Problem

• referring image segmentation
• parameter-efficient tuning
• non-target activation
• target-aware learning
• visual-text alignment

Innovation

• parameter-efficient tuning
• referring image segmentation
• non-target activation
• target-aware learning
• contrastive learning