Structured-Condensed Prompt Tuning in Vision-Language Models for Fine-grained Image Recognition

2026-07-07

AI Summary

Existing vision-language models struggle to capture subtle semantic distinctions among fine-grained categories, and conventional prompt tuning approaches often overlook their structured relational properties, thereby limiting discriminative capacity. To address this, this work proposes Semantic Relation Encoding (SRE) to explicitly model the topological structure of category relationships and introduces a Semantic Compression Loss (ScLoss) to distill highly discriminative semantic components, moving beyond the assumption of discrete labels. Integrating SRE and ScLoss into a CLIP-based prompt tuning framework, the method achieves state-of-the-art performance across 14 fine-grained recognition benchmarks, significantly alleviating semantic ambiguity in few-shot learning and cross-category generalization scenarios.

Abstract

Fine-grained image recognition poses a significant challenge due to the substantial expertise and effort required for manual annotation. Vision-language models (VLMs) like CLIP provide a compelling zero-shot alternative, reducing reliance on extensive labeled data. However, their ability to capture subtle distinctions remains limited, leading to subpar recognition performance. While prompt tuning has proven effective for adapting VLMs, most existing methods treat class labels as isolated, discrete entities, overlooking the rich semantic relationships between them. This oversimplified assumption limits the model's ability to capture hierarchical dependencies and inter-class correlations -- both critical for distinguishing visually similar categories. The problem is especially acute in fine-grained classification, where accurate recognition depends on understanding complex label semantics. To address this, we propose Structured-Condensed Prompt Tuning (SCPT), which enhances semantic structure modeling in prompt learning. Specifically, we introduce Semantic Relation Encoding (SRE) to explicitly model inter-class semantic topology and encode structured label relationships. In parallel, we design a Semantic Condensation loss (ScLoss) to suppress redundant supervision and extract discriminative components from the global semantic space. Together, these components significantly improve semantic alignment and fine-grained discrimination. Extensive experiments on 14 fine-grained benchmarks show that SCPT effectively mitigates semantic ambiguity and achieves state-of-the-art performance in both few-shot and base-to-novel generalization settings.

Problem

• fine-grained image recognition
• vision-language models
• semantic relationships
• prompt tuning
• class labels

Innovation

• Structured-Condensed Prompt Tuning
• Semantic Relation Encoding
• Semantic Condensation loss
• Vision-Language Models
• Fine-grained Image Recognition