Open-World Object Detection with Instance Representation Learning

2024-09-24

arXiv.org

AI Summary

This work addresses the neglect of semantic relationships among object instances in open-world object detection (OWOD). We propose the first unified framework jointly optimizing unknown-class detection and semantic-rich instance embedding learning. Methodologically: (1) we introduce a SAM-mask-guided unknown-box refinement module to improve localization accuracy for unknown instances; (2) we design a relaxed contrastive embedding transfer module based on vision foundation model (VFM) feature distillation, enabling cross-domain, transferable instance-level semantic similarity modeling. Experiments demonstrate significant improvements in both unknown-class detection performance and embedding quality on standard OWOD benchmarks. Moreover, the learned embeddings yield state-of-the-art results on downstream open-world tracking tasks, validating their generalizability and semantic fidelity.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Open World Object Detection(OWOD) addresses realistic scenarios where unseen object classes emerge, enabling detectors trained on known classes to detect unknown objects and incrementally incorporate the knowledge they provide. While existing OWOD methods primarily focus on detecting unknown objects, they often overlook the rich semantic relationships between detected objects, which are essential for scene understanding and applications in open-world environments (e.g., open-world tracking and novel class discovery). In this paper, we extend the OWOD framework to jointly detect unknown objects and learn semantically rich instance embeddings, enabling the detector to capture fine-grained semantic relationships between instances. To this end, we propose two modules that leverage the rich and generalizable knowledge of Vision Foundation Models(VFM). First, the Unknown Box Refine Module uses instance masks from the Segment Anything Model to accurately localize unknown objects. The Embedding Transfer Module then distills instance-wise semantic similarities from VFM features to the detector's embeddings via a relaxed contrastive loss, enabling the detector to learn a semantically meaningful and generalizable instance feature. Extensive experiments show that our method significantly improves both unknown object detection and instance embedding quality, while also enhancing performance in downstream tasks such as open-world tracking.

Problem

• Detects unknown objects in open-world scenarios
• Learns semantically rich instance embeddings
• Improves unknown object detection and embedding quality

Innovation

• Leverages Vision Foundation Models for semantic embeddings
• Uses Segment Anything Model for precise object localization
• Applies relaxed contrastive loss for embedding transfer