Geometry Matters: 3D Foundation Priors for Learning Semantic Correspondence

2026-05-28

AI Summary

Existing semantic correspondence methods based on 2D foundation models lack explicit 3D awareness, often confusing symmetric structures, repetitive parts, and visually similar regions that differ in spatial location. This work proposes a 3D-aware post-training framework that requires no manual pose annotations: it leverages SAM3D to automatically acquire instance-level 3D geometry and pose, refines geometric accuracy through render-and-compare optimization, and projects PartField descriptors onto the image plane to fuse with DINO and Stable Diffusion features for learning semantic correspondences. By incorporating precise geometric priors, the method eliminates the limitations of prior approaches that relied on coarse spherical assumptions and strong supervision, significantly improving correspondence accuracy and outperforming existing post-training techniques.

Abstract

Foundation features from self-supervised vision models and text-to-image diffusion models have proven effective for semantic correspondence estimation. However, because these features are learned primarily from 2D image objectives, they lack explicit 3D awareness and often confuse symmetric object sides, repeated parts, and visually similar structures that are distinct in 3D. We introduce a 3D-aware post-training framework that goes beyond available 2D foundation features by incorporating priors from 3D foundation models. Given an image, our method uses SAM3D to estimate object geometry and pose, and refines the pose through render-and-compare optimization. Subsequently, we render PartField descriptors from the reconstructed geometry into the image plane based on the estimated object pose. The resulting geometry-aware feature maps complement DINO and Stable Diffusion features, while geodesic distances on the reconstructed shapes enable reliable filtering of candidate correspondences. We use the filtered matches as supervision to train a lightweight adapter on top of DINO and Stable Diffusion for semantic correspondence. In contrast to prior post-training approaches that require pose annotations and rely on coarse spherical geometry, our method automatically obtains instance-specific 3D structure and uses it to guide correspondence learning. Experiments show that our approach improves semantic correspondence over the prior methods while reducing manual geometric supervision. Code and model can be found at https:/github.com/GenIntel/3D-SC.

Problem

• semantic correspondence
• 3D awareness
• foundation models
• geometric ambiguity
• 2D-3D discrepancy

Innovation

• 3D-aware
• semantic correspondence
• foundation models
• geometry prior
• render-and-compare optimization