From Local Matches to Global Masks: Novel Instance Detection in Open-World Scenes

2026-03-03

AI Summary

This work proposes L2G-Det, a novel framework for detecting and segmenting novel object instances in open-world scenarios using only a few template images, addressing challenges such as occlusion and background clutter. Departing from conventional proposal-based paradigms, L2G-Det introduces an end-to-end local-to-global pipeline: it generates candidate points through dense local patch matching between template and query images, then leverages these refined candidates to guide an enhanced Segment Anything Model (SAM) in reconstructing complete instance masks. Key innovations include the introduction of instance-level object tokens to strengthen SAM’s instance awareness and a tailored instance-specific prompting mechanism. Experiments demonstrate that L2G-Det significantly outperforms existing proposal-based methods in complex open-world settings, achieving markedly improved robustness and accuracy in novel instance detection and segmentation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Object Detection & Categorization
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Detecting and segmenting novel object instances in open-world environments is a fundamental problem in robotic perception. Given only a small set of template images, a robot must locate and segment a specific object instance in a cluttered, previously unseen scene. Existing proposal-based approaches are highly sensitive to proposal quality and often fail under occlusion and background clutter. We propose L2G-Det, a local-to-global instance detection framework that bypasses explicit object proposals by leveraging dense patch-level matching between templates and the query image. Locally matched patches generate candidate points, which are refined through a candidate selection module to suppress false positives. The filtered points are then used to prompt an augmented Segment Anything Model (SAM) with instance-specific object tokens, enabling reliable reconstruction of complete instance masks. Experiments demonstrate improved performance over proposal-based methods in challenging open-world settings.

Problem

• novel instance detection
• open-world scenes
• object segmentation
• template-based matching
• cluttered environments

Innovation

• local-to-global detection
• dense patch matching
• instance-specific prompting
• proposal-free segmentation
• open-world instance detection