Drag, Infer, Reproject: Grounding LLMs through Spatial Interaction for Image Clustering

2026-06-26

AI Summary

This work addresses the limitations of existing semantic clustering methods, which rely on predefined criteria and struggle to support users in dynamically forming and refining clustering intent during interaction. The authors propose a novel approach that integrates large language models with spatial drag-and-drop interactions: by interpreting users’ local image dragging actions, the system iteratively infers and refines their implicit clustering criteria in real time, driving global re-projection to generate layouts aligned with user intent. This is the first method to incorporate large language models into incremental semantic clustering, eliminating dependence on fixed priors and enabling human–AI collaborative discovery of dynamic clustering criteria and layout optimization. Experiments demonstrate that the system accurately extracts user goals from continuous interactions and progressively produces high-quality, intent-consistent clustering results.

Abstract

Dimension reduction and semantic interaction support image clustering by making similarity structure visible and manipulable. Existing semantic interaction methods encode users' clustering criterion (a user-interpretable semantic dimension, e.g., action, location, or mood) from direct manipulation to steer reprojection, giving users direct control over the resulting layout. Yet they typically depend on learned embeddings or a predefined criterion. In practice, users' clustering criterion often emerges gradually and becomes refined through interaction rather than being fully clear at the outset. In this work, we present CriterionSI (Criterion-guided Semantic Interaction), a method that translates incremental drag interactions into criterion-guided reprojection. CriterionSI uses large language models to infer and refine the clustering criterion from sequential user drags, while grounding semantic interpretation in human-provided feedback rather than fixed prior assumptions. CriterionSI combines the inferred criterion with local drags to guide global reprojection. The simulation-based evaluation and usage scenario demonstrate that CriterionSI can discover and refine the target criterion from sequential interactions and progressively produce criterion-aligned clustering layouts. Our code and data are available at: https://github.com/4C79/CriterionSI.

Problem

• image clustering
• semantic interaction
• clustering criterion
• spatial interaction
• user-driven refinement

Innovation

• semantic interaction
• large language models
• image clustering
• criterion inference
• reprojection