Rethinking Prototype-based Similarity Learning for Few-Shot Object Detection

2026-06-22

AI Summary

This work addresses the challenges of category confusion due to inter-class similarity and the lack of spatial localization in similarity matching within few-shot object detection. To this end, the authors propose Text-anchored Semantic Masks (TSMa) and a Stage-aligned Hierarchical Autoregressive Regression (SHARe) mechanism. TSMa leverages textual features as semantic anchors to suppress style-induced interference and enhance intrinsic class discriminability. SHARe formulates bounding box regression as a multi-stage progressive refinement process, aligning the abstraction capabilities of different Vision Transformer layers with corresponding regression stages to improve localization accuracy. Notably, the method generalizes to novel categories without additional training and achieves a new state-of-the-art performance on the COCO few-shot detection benchmark, surpassing the previous best approach by 10.1 nAP.

Abstract

Few-shot object detection aims to detect novel object categories from only a few labeled examples, avoiding costly large-scale annotation. Recent prototype-based similarity learning approaches enable training-free adaptation by matching query features with class prototypes. However, they suffer from two fundamental limitations: (i) class confusion arising from inter-class similarity margin collapse, and (ii) insufficient visual cues for precise localization, as similarity scores capture only class-level semantic affinity while providing limited spatial information. To address these issues, we introduce two complementary components. Text-Anchored Semantic Mask (TSMa) leverages class-level text features as semantic anchors to identify semantically aligned channels through channel-wise interaction between visual and text features. By suppressing style-induced spurious responses and emphasizing class-intrinsic signals, TSMa enlarges inter-class similarity margins and mitigates class confusion. We further propose Stage-Aligned Hierarchical Autoregressive Regression (SHARe), which reformulates localization as a hierarchical autoregressive process that progressively refines bounding boxes across multiple stages. SHARe leverages the layer-wise characteristics of ViT representations by aligning feature abstraction levels with regression stages: deeper layers guide early coarse localization, while shallower layers rich in edge and texture cues refine spatial details in later stages. Experiments on COCO demonstrate a new state of the art, outperforming the previous best by +10.1 nAP, with extensive analysis validating each component. The code is available at https://github.com/VisualScienceLab-KHU/ReSet.

Problem

• few-shot object detection
• prototype-based similarity learning
• class confusion
• localization
• semantic affinity

Innovation

• Prototype-based Similarity Learning
• Text-Anchored Semantic Mask
• Stage-Aligned Hierarchical Autoregressive Regression
• Few-Shot Object Detection
• Vision-Language Alignment