Unsupervised Collaborative Metric Learning with Mixed-Scale Groups for General Object Retrieval

2024-03-16

arXiv.org

AI Summary

This paper addresses open-vocabulary generic object retrieval, proposing MS-UGCML—an unsupervised collaborative metric learning framework—to tackle efficient multi-scale object matching and precise localization. Methodologically, it introduces a novel hybrid-scale grouping mechanism to model cross-scale semantic consistency; leverages SAM to extract spatial context; and integrates contrastive learning, multi-scale feature alignment, and collaborative prototype clustering for robust, annotation-free object-level embedding learning. Key contributions include: (1) the first open-vocabulary benchmark specifically designed for generic object retrieval; and (2) state-of-the-art performance across BelgaLogos, Visual Genome, LVIS, and a newly constructed test set—achieving up to 6.69% and 10.03% absolute gains in object-level and image-level mAP, respectively, significantly outperforming existing unsupervised approaches.

Technology Category

• Computer Vision: Image and Video Retrieval
• Machine Learning: Unsupervised & Self-Supervised Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

The task of searching for visual objects in a large image dataset is difficult because it requires efficient matching and accurate localization of objects that can vary in size. Although the segment anything model (SAM) offers a potential solution for extracting object spatial context, learning embeddings for local objects remains a challenging problem. This paper presents a novel unsupervised deep metric learning approach, termed unsupervised collaborative metric learning with mixed-scale groups (MS-UGCML), devised to learn embeddings for objects of varying scales. Following this, a benchmark of challenges is assembled by utilizing COCO 2017 and VOC 2007 datasets to facilitate the training and evaluation of general object retrieval models. Finally, we conduct comprehensive ablation studies and discuss the complexities faced within the domain of general object retrieval. Our object retrieval evaluations span a range of datasets, including BelgaLogos, Visual Genome, LVIS, in addition to a challenging evaluation set that we have individually assembled for open-vocabulary evaluation. These comprehensive evaluations effectively highlight the robustness of our unsupervised MS-UGCML approach, with an object level and image level mAPs improvement of up to 6.69% and 10.03%, respectively. The code is publicly available at https://github.com/dengyuhai/MS-UGCML.

Problem

• Enhancing visual question answering with object-level knowledge retrieval
• Learning embeddings for diverse, long-tailed objects at multiple scales
• Developing benchmarks for general object retrieval and OK-VQA evaluation

Innovation

• Unsupervised deep feature embedding for objects
• Multi-scale group collaborative embedding learning
• Object retrieval benchmark with diverse datasets