Personalized Object Identification and Localization via In-Context Inference with Vision-Language Models

2026-06-30

AI Summary

This work addresses a critical limitation in existing personalized object localization methods, which assume that every query image contains the target object and thus suffer from high false-positive rates when confronted with real-world negative samples lacking the target. To tackle this challenge, the paper introduces a new task—Personalized Object Identification and Localization (POIL)—and presents the first dedicated dataset for it. The authors propose IPLoc-ID, an algorithm that unifies object localization and instance verification within a single autoregressive framework. By integrating contextual reasoning from vision-language models, bounding box prediction, and a self-supervised query mechanism, IPLoc-ID effectively suppresses spurious detections while preserving high localization accuracy, thereby resolving the practical challenges of instance-level recognition and localization in realistic scenarios.

Abstract

Personalized object localization (POL) localizes an object instance in a query image based on a few reference images with bounding-box annotations and a target object label. The pioneering method, IPLoc, solves this task through in-context inference with vision-language models (VLMs). However, it assumes that the query image always contains the target object. This assumption severely limits its applicability to real-world scenarios with many irrelevant images. To address this issue, we formulate a new task, personalized object identification and localization (POIL), by positioning POL within the broader few-shot object detection framework. POIL aims to localize the target object instance while rejecting query images that do not contain the reference object instance. We also present POIL datasets constructed from public sources. We further propose an in-context algorithm named IPLoc-ID for solving POIL with VLMs. IPLoc-ID first predicts a candidate bounding box and then determines whether it corresponds to the reference object instance. We introduce a self-posed query to connect these two steps within a single autoregressive generation framework. Through ablation studies and comprehensive experiments, we show that IPLoc-ID substantially suppresses false-positive detections on negative query images while maintaining localization performance comparable to IPLoc. Overall, IPLoc-ID effectively addresses the practical instance-level POIL task, which cannot be sufficiently solved by conventional object detection, few-shot object detection, or the localization-only IPLoc method.

Problem

• Personalized Object Localization
• In-Context Inference
• Vision-Language Models
• Few-Shot Object Detection
• Object Identification

Innovation

• personalized object identification and localization
• in-context inference
• vision-language models
• few-shot object detection
• self-posed query