Synergistic Dual-Branch Adaptation for Multi-modal Generalized Category Discovery

2026-06-19

AI Summary

This work addresses key challenges in multimodal generalized category discovery—namely, coarse-grained cross-modal collaboration, textual encoding noise, and the absence of fine-grained relational supervision—by proposing a plug-and-play collaborative dual-branch adaptation framework. During encoding, a lightweight cross-modal adapter dynamically fuses visual information to enhance textual features, while a neighborhood mutual learning module aligns the local neighborhood distributions of both branches via bidirectional KL divergence, thereby enabling fine-grained relational supervision. Extensive experiments demonstrate that the proposed framework achieves state-of-the-art performance across six benchmark datasets and consistently improves upon various baseline methods, confirming its effectiveness and broad applicability.

Abstract

Generalized Category Discovery (GCD) aims to classify old categories and discover new ones from unlabeled data. Recent multi-modal approaches introduce retrieved or synthesized texts into a dual-branch architecture to provide semantic cues complementary to visual features. However, the cross-modal synergy in existing dual-branch methods remains coarse and incomplete: the two modalities are encoded independently with the bias and noise in the derived text left unaddressed during encoding, and existing mutual learning strategies operate only on global class-level anchors, lacking fine-grained relational supervision. To address these limitations, we propose the Synergistic Dual-Branch Adaptation (SDBA) framework, which serves as a plug-and-play enhancement compatible with existing dual-branch methods such as GET and TextGCD. SDBA comprises two components: the cross-modal synergistic adapter inserts lightweight adapters into both branches and further injects visual information into the text adapter at each encoder layer to enhance text feature learning during encoding; the neighborhood mutual learning module enforces consistent local neighborhood distributions between the two branches via bidirectional KL divergence, providing fine-grained relational supervision for both old and new classes. Extensive experiments on six benchmarks demonstrate state-of-the-art performance, and consistent improvements on different baselines validate the broad scalability of the proposed framework.

Problem

• Generalized Category Discovery
• multi-modal learning
• dual-branch architecture
• cross-modal synergy
• fine-grained relational supervision

Innovation

• Synergistic Dual-Branch Adaptation
• Cross-modal Synergy
• Neighborhood Mutual Learning
• Multi-modal Generalized Category Discovery
• Fine-grained Relational Supervision