Do It Yourself: Learning Semantic Correspondence from Pseudo-Labels

2025-06-05

AI Summary

This paper addresses the ambiguity in semantic keypoint matching for symmetric or repetitive objects in computer vision. To achieve robust self-supervised cross-image and cross-instance matching, we propose a 3D-aware pseudo-labeling framework. Our method introduces three key innovations: (1) a 3D-aware chained pseudo-label generation scheme that incorporates geometric priors to guide pseudo-label construction; (2) a relaxed cycle-consistency filtering mechanism to enhance robustness against labeling noise; and (3) a 3D spherical prototype mapping constraint to mitigate semantic ambiguity. Leveraging pre-trained visual features, we employ a lightweight adapter network jointly optimized with the matching objective. On SPair-71k, our approach surpasses the state-of-the-art by 4.0% absolute accuracy and outperforms weakly supervised methods by over 7%. We further demonstrate strong cross-dataset transferability, validating generalization across diverse data sources.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Semi-Supervised Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Finding correspondences between semantically similar points across images and object instances is one of the everlasting challenges in computer vision. While large pre-trained vision models have recently been demonstrated as effective priors for semantic matching, they still suffer from ambiguities for symmetric objects or repeated object parts. We propose to improve semantic correspondence estimation via 3D-aware pseudo-labeling. Specifically, we train an adapter to refine off-the-shelf features using pseudo-labels obtained via 3D-aware chaining, filtering wrong labels through relaxed cyclic consistency, and 3D spherical prototype mapping constraints. While reducing the need for dataset specific annotations compared to prior work, we set a new state-of-the-art on SPair-71k by over 4% absolute gain and by over 7% against methods with similar supervision requirements. The generality of our proposed approach simplifies extension of training to other data sources, which we demonstrate in our experiments.

Problem

• Improving semantic correspondence estimation using 3D-aware pseudo-labels
• Reducing ambiguities in symmetric or repeated object parts matching
• Minimizing need for dataset-specific annotations in semantic correspondence

Innovation

• 3D-aware pseudo-labeling for semantic correspondence
• Relaxed cyclic consistency filters wrong labels
• 3D spherical prototype mapping constraints