Universal Guideline-Driven Image Clustering via a Hybrid LLM Agent

2026-06-22

AI Summary

This work addresses the challenge that image clustering faces across diverse scenarios due to fundamental differences in task objectives. It proposes the first general-purpose image clustering framework that leverages textual instructions to guide the clustering process, enabling integration of complex semantic cues without requiring task-specific training. The approach combines generative concept proxy modeling, instruction-aware embeddings, a minimum spanning tree–based LLM traversal mechanism, and a selective reasoning strategy to achieve adaptive clustering across varied settings. Extensive experiments demonstrate its consistent superiority over existing specialized methods under multiple configurations—including generic versus fine-grained, global versus local, and balanced versus long-tailed distributions—thereby validating its generality and effectiveness.

Abstract

Unifying image clustering across different clustering scenarios remains challenging due to fundamental gaps among tasks. We introduce a Guideline-Driven Image Clustering Agent, the first universal framework that bridges these gaps through textual guidelines. To incorporate complex guidelines without task-specific training, we propose Generative Concept Proxy Modeling, which generates guideline-aware embeddings via concept proxy extraction. For scenarios requiring automatic cluster discovery, we introduce LLM Traversal based on Minimum Spanning Tree that selectively applies LLM reasoning for complex semantic judgments. Our method generalizes across diverse clustering scenarios spanning from general to fine-grained categorization, from global to local criteria, and from balanced to long-tail distributions. Our framework consistently outperforms specialized methods across diverse clustering tasks.

Problem

• image clustering
• universal framework
• clustering scenarios
• guideline-driven
• cluster discovery

Innovation

• Guideline-Driven Clustering
• Generative Concept Proxy Modeling
• LLM Traversal
• Universal Image Clustering
• Minimum Spanning Tree