VLM-NCD:Novel Class Discovery with Vision-Based Large Language Models

2025-12-10

AI Summary

Existing vision-based methods for Novel Class Discovery (NCD) in unlabeled data suffer from insufficient discriminability and poor robustness to long-tailed class distributions. Method: This paper proposes the first multimodal NCD framework integrating vision–text semantic priors: (1) joint modeling of image and text features to construct semantic prototypes and cluster centers; (2) a two-stage, semantic-affinity-driven adaptive clustering mechanism—novel in NCD—that achieves strong robustness to long-tailed distributions; and (3) prototype-guided clustering, semantic-affinity thresholding, and dynamic sample separation to enhance decoupling between known and unknown classes. Results: On CIFAR-100, our method achieves up to 25.3% higher unknown-class identification accuracy than state-of-the-art approaches, significantly mitigating performance degradation induced by long-tailed class imbalance.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Novel Class Discovery aims to utilise prior knowledge of known classes to classify and discover unknown classes from unlabelled data. Existing NCD methods for images primarily rely on visual features, which suffer from limitations such as insufficient feature discriminability and the long-tail distribution of data. We propose LLM-NCD, a multimodal framework that breaks this bottleneck by fusing visual-textual semantics and prototype guided clustering. Our key innovation lies in modelling cluster centres and semantic prototypes of known classes by jointly optimising known class image and text features, and a dualphase discovery mechanism that dynamically separates known or novel samples via semantic affinity thresholds and adaptive clustering. Experiments on the CIFAR-100 dataset show that compared to the current methods, this method achieves up to 25.3% improvement in accuracy for unknown classes. Notably, our method shows unique resilience to long tail distributions, a first in NCD literature.

Problem

• Discovers unknown classes using known class knowledge
• Overcomes visual feature limitations in image classification
• Improves accuracy for novel classes in long-tail distributions

Innovation

• Fuses visual-textual semantics for enhanced discriminability
• Uses prototype guided clustering to model known class centers
• Implements dual-phase discovery with adaptive semantic thresholds