Geometry-Aware CLIP Retrieval via Local Cross-Modal Alignment and Steering

2026-04-13

AI Summary

This work addresses the limitations of CLIP-based retrieval, particularly misranking of neighboring items and weak controllability—such as confusion among fine-grained attributes—stemming from local geometric inconsistencies. To tackle these issues, the paper reframes retrieval as a neighborhood alignment task and introduces a novel inference-time paradigm that requires no retraining. The approach leverages the Hungarian algorithm for neighborhood-level reranking and incorporates a query-conditioned local guidance mechanism that exploits directional signals derived from contrastive learning to refine the retrieval structure. By synergistically combining reranking with local guidance—two complementary strategies—the method substantially improves performance on attribute binding and compositional retrieval tasks, effectively mitigating systematic geometric errors and enhancing both relevance and controllability of retrieval results.

Technology Category

• Computer Vision: Image and Video Retrieval
• Search and Optimization: Local Search
• Machine Learning: Learning Preferences or Rankings

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

CLIP retrieval is typically framed as a pointwise similarity problem in a shared embedding space. While CLIP achieves strong global cross-modal alignment, many retrieval failures arise from local geometric inconsistencies: nearby items are incorrectly ordered, leading to systematic confusions (e.g., pentagon vs. hexagon) and produces diffuse, weakly controlled result sets. Prior work largely optimizes for point wise relevance or finetuning to mitigate these problems. We instead view retrieval as a problem of neighborhood alignment. Our work introduces (1) neighborhood-level re-ranking via Hungarian matching, which rewards structural consistency; (2) query-conditioned local steering, where directions derived from contrastive neighborhoods around the query reshape retrieval. We show that these techniques improve retrieval performance on attribute-binding and compositional retrieval tasks. Together, these methods operate on local neighborhoods but serve different roles: re-ranking rewards alignment whereas local steering controls neighborhood structure. This shows that retrieval quality and controllability depend critically on local structure, which can be exploited at inference time without retraining.

Problem

• CLIP retrieval
• local geometric inconsistency
• neighborhood alignment
• attribute-binding
• compositional retrieval

Innovation

• neighborhood alignment
• local steering
• Hungarian matching
• geometry-aware retrieval
• CLIP