IntRec: Intent-based Retrieval with Contrastive Refinement

2026-02-19

AI Summary

This work addresses the challenge of object retrieval in complex scenes, where user queries often suffer from semantic ambiguity or high similarity among multiple targets, and existing open-vocabulary detectors lack the ability to refine results through user feedback. To this end, we propose IntRec, an interactive object retrieval framework that models user intent states, maintains positive and negative memory sets, and employs a contrastive alignment function to dynamically refine candidate rankings for fine-grained disambiguation. Notably, IntRec requires no additional supervision and achieves efficient, low-latency interaction (<30 ms per round). On the LVIS benchmark, it attains 35.4 AP, outperforming OVMR, CoDet, and CAKE. Moreover, on the LVIS-Ambiguous benchmark, a single round of feedback yields a 7.9 AP improvement, substantially enhancing retrieval accuracy in ambiguous scenarios.

Technology Category

• Computer Vision: Image and Video Retrieval
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Retrieving user-specified objects from complex scenes remains a challenging task, especially when queries are ambiguous or involve multiple similar objects. Existing open-vocabulary detectors operate in a one-shot manner, lacking the ability to refine predictions based on user feedback. To address this, we propose IntRec, an interactive object retrieval framework that refines predictions based on user feedback. At its core is an Intent State (IS) that maintains dual memory sets for positive anchors (confirmed cues) and negative constraints (rejected hypotheses). A contrastive alignment function ranks candidate objects by maximizing similarity to positive cues while penalizing rejected ones, enabling fine-grained disambiguation in cluttered scenes. Our interactive framework provides substantial improvements in retrieval accuracy without additional supervision. On LVIS, IntRec achieves 35.4 AP, outperforming OVMR, CoDet, and CAKE by +2.3, +3.7, and +0.5, respectively. On the challenging LVIS-Ambiguous benchmark, it improves performance by +7.9 AP over its one-shot baseline after a single corrective feedback, with less than 30 ms of added latency per interaction.

Problem

• object retrieval
• ambiguous queries
• complex scenes
• open-vocabulary detection
• user feedback

Innovation

• interactive object retrieval
• intent state
• contrastive alignment
• open-vocabulary detection
• user feedback refinement