CLAY: Conditional Visual Similarity Modulation in Vision-Language Embedding Space

2026-04-13

AI Summary

This work addresses the limitations of traditional image retrieval systems, which rely on fixed similarity metrics and struggle to accommodate users’ diverse multi-condition queries. To overcome this, we propose CLAY, a method that leverages pre-trained vision-language models to dynamically modulate the embedding space through textual conditions—without requiring additional training—enabling efficient and flexible multi-condition image retrieval. CLAY innovatively decouples textual conditions from visual features, allowing similarity measures to be flexibly reconfigured atop fixed visual embeddings. To evaluate such capabilities, we introduce CLAY-EVAL, the first synthetic benchmark tailored for multi-condition retrieval. Experiments demonstrate that CLAY significantly outperforms existing approaches on both standard datasets and CLAY-EVAL, achieving high retrieval accuracy while substantially improving computational efficiency.

Technology Category

• Computer Vision: Image and Video Retrieval
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Human perception of visual similarity is inherently adaptive and subjective, depending on the users' interests and focus. However, most image retrieval systems fail to reflect this flexibility, relying on a fixed, monolithic metric that cannot incorporate multiple conditions simultaneously. To address this, we propose CLAY, an adaptive similarity computation method that reframes the embedding space of pretrained Vision-Language Models (VLMs) as a text-conditional similarity space without additional training. This design separates the textual conditioning process and visual feature extraction, allowing highly efficient and multi-conditioned retrieval with fixed visual embeddings. We also construct a synthetic evaluation dataset CLAY-EVAL, for comprehensive assessment under diverse conditioned retrieval settings. Experiments on standard datasets and our proposed dataset show that CLAY achieves high retrieval accuracy and notable computational efficiency compared to previous works.

Problem

• visual similarity
• image retrieval
• conditional retrieval
• vision-language models
• adaptive similarity

Innovation

• conditional similarity
• vision-language embedding
• adaptive retrieval
• text-conditioned modulation
• zero-shot adaptation