Balancing Conservatism and Aggressiveness: Prototype-Affinity Hybrid Network for Few-Shot Segmentation

2025-07-25

AI Summary

In few-shot segmentation (FSS), existing methods face a fundamental trade-off: prototype-based learning tends to be overly conservative, while affinity-based learning suffers from erroneous foreground-background matching. To address this, we propose PAHNet—a dual-paradigm collaborative framework unifying prototype and affinity learning. Its core innovations are the Prototype-Guided Feature Enhancement (PFE) module, which dynamically modulates query features using support-set prototypes, and the Attention Score Calibration (ASC) module, which reweights foreground-background affinities under prototype constraints to suppress mismatches. This design jointly enhances discriminability without compromising generalizability. Extensive experiments on PASCAL-5i and COCO-20i demonstrate that PAHNet consistently outperforms state-of-the-art methods under both 1-shot and 5-shot settings, validating the effectiveness and robust generalization of its balanced dual-paradigm strategy.

Technology Category

• Computer Vision: Segmentation
• Search and Optimization: Learning to Search
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

This paper studies the few-shot segmentation (FSS) task, which aims to segment objects belonging to unseen categories in a query image by learning a model on a small number of well-annotated support samples. Our analysis of two mainstream FSS paradigms reveals that the predictions made by prototype learning methods are usually conservative, while those of affinity learning methods tend to be more aggressive. This observation motivates us to balance the conservative and aggressive information captured by these two types of FSS frameworks so as to improve the segmentation performance. To achieve this, we propose a **P**rototype-**A**ffinity **H**ybrid **Net**work (PAHNet), which introduces a Prototype-guided Feature Enhancement (PFE) module and an Attention Score Calibration (ASC) module in each attention block of an affinity learning model (called affinity learner). These two modules utilize the predictions generated by a pre-trained prototype learning model (called prototype predictor) to enhance the foreground information in support and query image representations and suppress the mismatched foreground-background (FG-BG) relationships between them, respectively. In this way, the aggressiveness of the affinity learner can be effectively mitigated, thereby eventually increasing the segmentation accuracy of our PAHNet method. Experimental results show that PAHNet outperforms most recently proposed methods across 1-shot and 5-shot settings on both PASCAL-5$^i$ and COCO-20$^i$ datasets, suggesting its effectiveness. The code is available at: [GitHub - tianyu-zou/PAHNet: Balancing Conservatism and Aggressiveness: Prototype-Affinity Hybrid Network for Few-Shot Segmentation (ICCV'25)](https://github.com/tianyu-zou/PAHNet)

Problem

• Balancing conservative and aggressive predictions in few-shot segmentation
• Improving segmentation accuracy using hybrid prototype-affinity learning
• Enhancing foreground information while suppressing mismatched FG-BG relationships

Innovation

• Hybrid network balances prototype and affinity learning
• Prototype-guided Feature Enhancement module improves foreground
• Attention Score Calibration suppresses mismatched FG-BG relationships