Hierarchical Textual Knowledge for Enhanced Image Clustering

2026-04-13

AI Summary

This work addresses the limitation of traditional image clustering methods, which rely solely on visual features and struggle to distinguish semantically distinct yet visually similar categories. To overcome this, the authors propose a novel unsupervised clustering framework that incorporates hierarchical textual knowledge for the first time. By leveraging large language models with structured prompts, the method extracts abstract concepts and their discriminative attributes, constructs knowledge-enhanced features, and fuses them with original visual representations—all without requiring any training. Evaluated across 20 benchmark datasets, the approach significantly outperforms existing clustering methods, surpassing zero-shot CLIP on 14 of them. Moreover, it avoids performance degradation caused by improper textual usage, thereby enhancing both semantic discriminability and robustness.

Technology Category

• Machine Learning: Clustering
• Computer Vision: Representation Learning for Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Image clustering aims to group images in an unsupervised fashion. Traditional methods focus on knowledge from visual space, making it difficult to distinguish between visually similar but semantically different classes. Recent advances in vision-language models enable the use of textual knowledge to enhance image clustering. However, most existing methods rely on coarse class labels or simple nouns, overlooking the rich conceptual and attribute-level semantics embedded in textual space. In this paper, we propose a knowledge-enhanced clustering (KEC) method that constructs a hierarchical concept-attribute structured knowledge with the help of large language models (LLMs) to guide clustering. Specifically, we first condense redundant textual labels into abstract concepts and then automatically extract discriminative attributes for each single concept and similar concept pairs, via structured prompts to LLMs. This knowledge is instantiated for each input image to achieve the knowledge-enhanced features. The knowledge-enhanced features with original visual features are adapted to various downstream clustering algorithms. We evaluate KEC on 20 diverse datasets, showing consistent improvements across existing methods using additional textual knowledge. KEC without training outperforms zero-shot CLIP on 14 out of 20 datasets. Furthermore, the naive use of textual knowledge may harm clustering performance, while KEC provides both accuracy and robustness.

Problem

• image clustering
• textual knowledge
• semantic ambiguity
• hierarchical semantics
• vision-language models

Innovation

• hierarchical textual knowledge
• knowledge-enhanced clustering
• vision-language models
• large language models
• unsupervised image clustering