Relational Visual Similarity

2025-12-08

AI Summary

Existing visual similarity metrics (e.g., LPIPS, CLIP, DINO) capture only perceptual attribute similarity and fail to model rich relational analogies—such as functional equivalence or structural correspondence—that underpin human cognition. Method: This work pioneers the formulation of image relational similarity as a learnable task. We introduce a large-scale, manually annotated dataset of 114K image pairs with logical relational labels and propose a novel fine-tuning paradigm based on identity-agnostic image–text pairs to steer vision–language models toward aligning latent relational structures across visually dissimilar images. Contribution/Results: Experiments demonstrate substantial gains in relational similarity discrimination, exposing fundamental cognitive limitations of mainstream vision models. Our approach advances visual representation learning from “perceptually discriminative” toward “cognitively interpretable,” thereby bridging a critical gap in relational perception computing.

Technology Category

• Computer Vision: Language and Vision
• Cognitive Modeling & Cognitive Systems: Analogy
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Humans do not just see attribute similarity -- we also see relational similarity. An apple is like a peach because both are reddish fruit, but the Earth is also like a peach: its crust, mantle, and core correspond to the peach's skin, flesh, and pit. This ability to perceive and recognize relational similarity, is arguable by cognitive scientist to be what distinguishes humans from other species. Yet, all widely used visual similarity metrics today (e.g., LPIPS, CLIP, DINO) focus solely on perceptual attribute similarity and fail to capture the rich, often surprising relational similarities that humans perceive. How can we go beyond the visible content of an image to capture its relational properties? How can we bring images with the same relational logic closer together in representation space? To answer these questions, we first formulate relational image similarity as a measurable problem: two images are relationally similar when their internal relations or functions among visual elements correspond, even if their visual attributes differ. We then curate 114k image-caption dataset in which the captions are anonymized -- describing the underlying relational logic of the scene rather than its surface content. Using this dataset, we finetune a Vision-Language model to measure the relational similarity between images. This model serves as the first step toward connecting images by their underlying relational structure rather than their visible appearance. Our study shows that while relational similarity has a lot of real-world applications, existing image similarity models fail to capture it -- revealing a critical gap in visual computing.

Problem

• Develops a metric for relational similarity between images
• Creates a dataset to train models on relational logic
• Addresses the gap in current visual similarity models

Innovation

• Anonymized captions describe relational logic not surface content
• Fine-tuned Vision-Language model measures relational similarity between images
• Dataset enables linking images by underlying relational structure not appearance