Fine-grained CLIP fine-tuning with self-annotated region alignment

2026-07-15

AI Summary

This work addresses the limitations of CLIP in fine-grained dense feature representation and the reliance of existing fine-tuning approaches on additional region-level annotations, which compromises global semantic consistency. The authors propose SFF-CLIP, a method that leverages only image–text pairs to enhance fine-grained representations without manual annotations. During fine-tuning, SFF-CLIP automatically generates concept phrases and dynamically aligns them with region features guided by attention heatmaps. This approach significantly improves performance on dense understanding tasks while preserving CLIP’s original global vision–language alignment capability, maintaining competitive results on image-level tasks comparable to the original CLIP model.

Abstract

Contrastive Language-Image Pre-training (CLIP) has been shown to have limitations in its fine-grained dense feature representation, due to its pre-training focusing on matching the whole image to a text description. Considering the large data and computational burden in pre-training a vision-language model from scratch, a series of works aim to enhance the fine-grained ability of CLIP through a fine-tuning scheme. However, existing works suffer from a variety of limitations: additional region annotations are usually required, which limits the semantic diversity due to the predefined categories and leads to a large effort to process the training data; and they usually sacrifice CLIP's original ability for global visual representation. To bypass these limitations, we propose SFF-CLIP (Self-annotated Fine-grained Fine-tuning for CLIP), which only uses image-text pairs as input to boost the fine-grained representation ability in the CLIP fine-tuning, while maintaining the global visual-semantic consistency. Concretely, a run-time region-phrase alignment scheme is designed, which obtains concept phrases from the input sentence, and aligns them with corresponding extracted region-based features using text-specific heat maps. Extensive experiments demonstrate that SFF-CLIP leads to significant performance improvements on fine-grained dense feature representation, as well as maintaining the performance of the original CLIP on image-level tasks. Code will be released later.

Problem

• fine-grained representation
• CLIP fine-tuning
• region annotation
• vision-language model
• global visual-semantic consistency

Innovation

• fine-grained representation
• CLIP fine-tuning
• self-annotation
• region-phrase alignment
• vision-language model