Learning Deep Modality-Shared Self-Expressiveness for Image Clustering with Textual Information

2026-08-08

AI Summary

Existing multimodal image clustering methods often compromise modality-specific structures by directly aligning cross-modal representations, leading to unreliable alignment. To address this issue, this work proposes DeepMORSE, which introduces a novel modality-shared self-expressive mechanism that preserves the intrinsic subspace structure of each modality while enabling effective cross-modal alignment. Theoretically, this mechanism suppresses inter-class noise and promotes subspace-preserving solutions, with mini-batch optimization implicitly inducing a regularizing effect. Leveraging deep representations from vision-language models and self-expressive learning, DeepMORSE achieves over 3% absolute improvement in clustering performance across six benchmarks—including UCF-101, DTD-47, and ImageNet-Dogs—and attains state-of-the-art results in image retrieval and zero-shot classification without task-specific losses or post-processing.

Abstract

Leveraging textual information for image clustering has emerged as a promising direction, largely owing to the powerful representations learned by Vision-Language Models (VLMs). Existing approaches typically retrieve a textual counterpart for each image and then refine multimodal representations by directly enforcing cross-modal agreement, e.g., maximizing image-text similarity inherited from pretrained VLMs. However, such a strategy aligns heterogeneous representations across modalities without explicitly modeling the intrinsic structure within each modality and thus might yield unreliable alignment or distort modality-specific structures that are crucial for clustering. In this paper, we propose a simple but principled approach, termed deep modality-shared self-expressive model (DeepMORSE), which discovers cross-modal structures via a modality-shared self-expressive model and simultaneously learns structured representations that conform to a union of modality-specific subspaces. Moreover, we theoretically justify that the modality-shared self-expressive coefficients suppress inter-class noise towards a subspace-preserving solution, and show that mini-batch optimization procedure introduces an implicit regularization onto the self-expressive model. We evaluate our DeepMORSE on six widely used image clustering benchmarks and observe performance improvements exceeding 3% on the UCF-101, DTD-47, and ImageNet-Dogs datasets. In addition, we demonstrate the strong transferability of the learned representations by achieving state-of-the-art performance on downstream tasks such as image retrieval and zero-shot classification---without requiring any task-specific losses or post-processing. The code is available at: https://github.com/mengxianghan123/DeepMORSE.

Problem

• image clustering
• textual information
• cross-modal alignment
• modality-specific structure
• Vision-Language Models

Innovation

• modality-shared self-expressiveness
• image clustering
• vision-language models
• structured representation learning
• subspace-preserving