Shape-of-You: Fused Gromov-Wasserstein Optimal Transport for Semantic Correspondence in-the-Wild

2026-03-12

AI Summary

This work addresses the ambiguity in semantic correspondence arising from geometric symmetries and repetitive structures in unlabeled in-the-wild images. To this end, we propose a novel approach that integrates the Gromov-Wasserstein optimal transport framework—introduced here for the first time to semantic correspondence—to jointly optimize feature similarity and structural consistency. Leveraging geometric priors from 3D foundation models, our method employs an anchor-based linearization approximation to reduce computational complexity and incorporates a soft-target loss to mitigate the effects of noisy supervision. By fusing 2D and 3D features, the proposed method achieves state-of-the-art performance on SPair-71k and AP-10k benchmarks, yielding highly accurate correspondences without requiring explicit geometric annotations.

Technology Category

• Computer Vision: Other Foundations of Computer Vision
• Search and Optimization: Sampling/Simulation-based Search
• Constraint Satisfaction and Optimization: Other Foundations of Constraint Satisfaction

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Semantic correspondence is essential for handling diverse in-the-wild images lacking explicit correspondence annotations. While recent 2D foundation models offer powerful features, adapting them for unsupervised learning via nearest-neighbor pseudo-labels has key limitations: it operates locally, ignoring structural relationships, and consequently its reliance on 2D appearance fails to resolve geometric ambiguities arising from symmetries or repetitive features. In this work, we address this by reformulating pseudo-label generation as a Fused Gromov-Wasserstein (FGW) problem, which jointly optimizes inter-feature similarity and intra-structural consistency. Our framework, Shape-of-You (SoY), leverages a 3D foundation model to define this intra-structure in the geometric space, resolving abovementioned ambiguity. However, since FGW is a computationally prohibitive quadratic problem, we approximate it through anchor-based linearization. The resulting probabilistic transport plan provides a structurally consistent but noisy supervisory signal. Thus, we introduce a soft-target loss dynamically blending guidance from this plan with network predictions to build a learning framework robust to this noise. SoY achieves state-of-the-art performance on SPair-71k and AP-10k datasets, establishing a new benchmark in semantic correspondence without explicit geometric annotations. Code is available at Shape-of-You.

Problem

• semantic correspondence
• in-the-wild images
• geometric ambiguity
• unsupervised learning
• structural consistency

Innovation

• Fused Gromov-Wasserstein
• semantic correspondence
• 3D foundation model
• optimal transport
• unsupervised learning