GET: Unlocking the Multi-modal Potential of CLIP for Generalized Category Discovery

2024-03-15

arXiv.org

AI Summary

Generalized Category Discovery (GCD) aims to jointly classify unlabeled data containing both known and unknown categories, yet existing methods rely solely on visual features and suffer from limited discriminability among visually similar classes. To address this, we propose the first text-modality-augmented GCD framework. Our method introduces a Text Embedding Synthesizer (TES) that inversely maps CLIP visual features into semantically plausible pseudo-text embeddings. We further design a dual-branch joint contrastive learning framework with an instance-level cross-modal consistency constraint to enable deep interaction and fusion between visual and pseudo-text modalities. Leveraging CLIP’s pre-trained multimodal alignment prior, our approach requires no real text annotations. Extensive experiments demonstrate significant improvements over state-of-the-art methods across all standard GCD benchmarks. The code is publicly available.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Given unlabelled datasets containing both old and new categories, generalized category discovery (GCD) aims to accurately discover new classes while correctly classifying old classes. Current GCD methods only use a single visual modality of information, resulting in a poor classification of visually similar classes. As a different modality, text information can provide complementary discriminative information, which motivates us to introduce it into the GCD task. However, the lack of class names for unlabelled data makes it impractical to utilize text information. To tackle this challenging problem, in this paper, we propose a Text Embedding Synthesizer (TES) to generate pseudo text embeddings for unlabelled samples. Specifically, our TES leverages the property that CLIP can generate aligned vision-language features, converting visual embeddings into tokens of the CLIP's text encoder to generate pseudo text embeddings. Besides, we employ a dual-branch framework, through the joint learning and instance consistency of different modality branches, visual and semantic information mutually enhance each other, promoting the interaction and fusion of visual and text knowledge. Our method unlocks the multi-modal potentials of CLIP and outperforms the baseline methods by a large margin on all GCD benchmarks, achieving new state-of-the-art. Our code is available at: https://github.com/enguangW/GET.

Problem

• Discover new classes and classify old ones in unlabelled datasets
• Improve classification of visually similar classes using text information
• Generate pseudo text embeddings for unlabelled data via CLIP

Innovation

• Generates pseudo text embeddings via CLIP
• Employs dual-branch framework for multi-modal fusion
• Converts visual embeddings to text tokens