Organizing Unstructured Image Collections using Natural Language

2024-10-07

arXiv.org

AI Summary

This paper addresses semantic organization of unlabeled image collections by introducing the novel task of *open-vocabulary semantic multi-cluster discovery*: automatically discovering interpretable natural-language clustering criteria directly from images—without predefined textual prompts—and disentangling multiple underlying semantic substructures. To this end, we propose X-Cluster, a framework that leverages text as a proxy to jointly model image representations, generate language-based criteria, and hierarchically optimize multi-cluster structures—integrating multimodal large model (MLLM) encoding, cross-modal contrastive learning, and self-supervised semantic grouping. We construct two new benchmarks—COCO-4c and Food-4c—on which X-Cluster significantly outperforms existing methods. We further validate its utility in bias detection and social media image popularity analysis. All code and benchmarks are publicly released to advance research in interpretable visual clustering.

Technology Category

• Machine Learning: Clustering
• Computer Vision: Multi-modal Vision
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Organizing unstructured visual data into semantic clusters is a key challenge in computer vision. Traditional deep clustering approaches focus on a single partition of data, while multiple clustering (MC) methods address this limitation by uncovering distinct clustering solutions. The rise of large language models (LLMs) and multimodal LLMs has enhanced MC by allowing users to define text clustering criteria. However, expecting users to manually define such criteria for large datasets before understanding the data is impractical. In this work, we introduce the task of Open-ended Semantic Multiple Clustering, that aims to automatically discover clustering criteria from large, unstructured image collections, uncovering interpretable substructures without requiring human input. Our framework, X-Cluster: eXploratory Clustering, uses text as a proxy to concurrently reason over large image collections, discover partitioning criteria, expressed in natural language, and reveal semantic substructures. To evaluate X-Cluster, we introduce the COCO-4c and Food-4c benchmarks, each containing four grouping criteria and ground-truth annotations. We apply X-Cluster to various real-world applications, such as discovering biases and analyzing social media image popularity, demonstrating its utility as a practical tool for organizing large unstructured image collections and revealing novel insights. We will open-source our code and benchmarks for reproducibility and future research.

Problem

• Automatically discover clustering criteria from unstructured image collections.
• Uncover interpretable substructures without requiring human input.
• Organize large unstructured image collections using natural language.

Innovation

• Open-ended Semantic Multiple Clustering for images
• X-Cluster uses text for image clustering criteria
• Automated discovery of interpretable image substructures