Learning Cross-View Semantic Priors for Single-Reference Unseen Object Pose Estimation

2026-06-20

AI Summary

This work addresses the ambiguity in correspondence estimation for 6D object pose estimation of unseen objects from a single reference view, which arises due to insufficient cross-view semantic information. To this end, the authors propose a correspondence learning framework centered on early cross-view semantic priors. The method enhances semantic context fusion between query and reference views through a cross-view semantic interaction mechanism and jointly optimizes semantic matching and geometric consistency via two training constraints: an intra-view structure preservation (IVSP) loss and a reference-anchored geometric consistency (RAGC) loss. Leveraging features from vision foundation models, the final pose is obtained using weighted SVD. The approach achieves state-of-the-art performance across six benchmark datasets, significantly outperforming existing methods—particularly under challenging view-pair settings—while maintaining comparable inference speed.

Abstract

Single-reference unseen object 6D pose estimation reduces object onboarding by estimating poses of arbitrary novel objects from only one reference view. Recent correspondence-based pipelines have achieved robust performance with vision foundation model (VFM) features. However, they typically treat these features as intra-view descriptors, leaving dense visual-semantic cues, including appearance, structure, and context, insufficiently exchanged across views before geometric decoding. Consequently, the decoded point features may lack joint semantic and geometric discriminability, making correspondence estimation still difficult in challenging cases. Instead of processing features independently, we build the correspondence pipeline around an early cross-view semantic prior. Specifically, cross-view semantic interaction (CVSI) enables dense query and reference VFM tokens to exchange semantic context and form a cross-view prior. Nevertheless, direct CVSI may disturb the VFM token structure, while the resulting semantic prior still needs 3D representation consistency for rigid correspondence. To make this CVSI prior reliable for 3D correspondence learning, we introduce two complementary training-time constraints: the intra-view structure preservation (IVSP) loss preserves the original intra-view token affinity structure during interaction, while the reference-anchored geometric consistency (RAGC) loss enforces spatial representation consistency of decoded point features. The final pose is recovered from learned correspondences through weighted SVD. We further construct a challenging view-pair protocol from the BOP Challenge datasets YCB-V and TUD-L to evaluate robustness in difficult matching scenarios. Extensive experiments on six benchmarks under different view-pair settings show that our method achieves state-of-the-art performance while maintaining comparable inference speed.

Problem

• unseen object pose estimation
• cross-view semantic priors
• 6D pose estimation
• correspondence learning
• vision foundation models

Innovation

• cross-view semantic prior
• correspondence learning
• vision foundation model
• 6D pose estimation
• geometric consistency