CLIP meets DINO for Tuning Zero-Shot Classifier using Unlabeled Image Collections

2024-11-28

arXiv.org

AI Summary

CLIP exhibits insufficient discriminative power in fine-grained image classification, while self-supervised vision models (e.g., DINO) rely on fully supervised linear probing for downstream adaptation. Method: We propose NoLA—a label-free framework that synergistically integrates DINO’s robust visual representations with fine-grained semantic text descriptions generated by large language models (LLMs). It employs a pseudo-label-driven cross-modal alignment module to jointly optimize prompt tuning of CLIP’s visual encoder, augmented by DINO-based auxiliary supervision. Contribution/Results: NoLA establishes the first label-free prompt-tuning paradigm, unifying self-supervised visual features and LLM-derived textual knowledge to enhance CLIP. Evaluated on 11 standard image classification benchmarks, it achieves an average absolute accuracy gain of 3.6%, significantly outperforming state-of-the-art label-free methods (e.g., LaFTer). The code and models are publicly released.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

In the era of foundation models, CLIP has emerged as a powerful tool for aligning text&visual modalities into a common embedding space. However, the alignment objective used to train CLIP often results in subpar visual features for fine-grained tasks. In contrast, SSL-pretrained models like DINO excel at extracting rich visual features due to their specialized training paradigm. Yet, these SSL models require an additional supervised linear probing step, which relies on fully labeled data which is often expensive and difficult to obtain at scale. In this paper, we propose a label-free prompt-tuning method that leverages the rich visual features of self-supervised learning models (DINO) and the broad textual knowledge of large language models (LLMs) to largely enhance CLIP-based image classification performance using unlabeled images. Our approach unfolds in three key steps: (1) We generate robust textual feature embeddings that more accurately represent object classes by leveraging class-specific descriptions from LLMs, enabling more effective zero-shot classification compared to CLIP's default name-specific prompts. (2) These textual embeddings are then used to produce pseudo-labels to train an alignment module that integrates the complementary strengths of LLM description-based textual embeddings&DINO's visual features. (3) Finally, we prompt-tune CLIP's vision encoder through DINO-assisted supervision using the trained alignment module. This three-step process allows us to harness the best of visual&textual foundation models, resulting in a powerful and efficient approach that surpasses state-of-the-art label-free classification methods. Notably, our framework, NoLA (No Labels Attached), achieves an average absolute gain of 3.6% over the state-of-the-art LaFTer across 11 diverse image classification datasets. Our code&models can be found at https://github.com/fazliimam/NoLA.

Problem

• Enhance CLIP-based image classification using unlabeled images.
• Leverage DINO's visual features and LLMs' textual knowledge.
• Improve zero-shot classification without labeled data.

Innovation

• Leverages DINO's visual features and LLM's textual knowledge
• Generates robust textual embeddings using class-specific LLM descriptions
• Prompt-tunes CLIP's vision encoder with DINO-assisted supervision