More Than Where You Are: Learning Semantics, Structure, and Geometry from Cross-View Localization

2026-07-14

AI Summary

This work addresses the challenge of maintaining semantic, structural, and geometric consistency under extreme viewpoint variations in cross-view localization. The authors propose CROSS, a novel framework that reframes cross-view localization as a joint learning task beyond pose estimation, integrating 3D grounding alignment, structure-aware matching, and relative hypothesis ranking to cohesively model semantic, structural, and geometric consistency. Notably, CROSS introduces structure learning as an intrinsic constraint to preserve semantic integrity—avoiding the pitfalls of point-wise matching—and leverages the powerful 2D representation capabilities of vision foundation models to enhance geometric reasoning. Evaluated on KITTI and VIGOR benchmarks, the method achieves state-of-the-art performance, significantly improving semantic stability, structural reliability, and geometric transferability across drastically different viewpoints.

Abstract

Consistent cross-view understanding under extreme viewpoint changes is essential for spatial intelligence, as it enables models to recognize the same scene across extreme viewpoint gaps. Cross-view localization naturally provides a promising pathway toward this ability, as it requires a model to align ground-view imagery with geo-referenced satellite-view imagery despite drastic appearance changes to estimate camera poses. Recent visual foundation models have made this long-standing localization problem increasingly feasible by providing rich 2D representations for cross-view matching. However, we argue that cross-view localization should not be viewed merely as 2D matching or pose estimation. In this work, we revisit cross-view localization as more than pose estimation and investigate how it can help the model develop consistent cross-view understanding under extreme viewpoint changes, including stable semantics, reliable structure, and transferable geometry. We identify three key limitations of existing methods that prevent them from achieving this. They usually lack explicit 3D grounding, rely on strict point-wise matching that can weaken semantic consistency, and learn from an absolute objective that provides limited guidance for geometric reasoning. To address these limitations, we propose CROSS, a unified cross-view localization framework built upon 3D-grounded alignment, structure-aware matching, and hypothesis ranking. This formulation makes structure learning an intrinsic requirement, encourages semantic representations to remain stable, and enables the model to acquire transferable geometry. Extensive experiments on the KITTI and VIGOR datasets show that CROSS achieves state-of-the-art performance in cross-view localization. More importantly, CROSS effectively learns stable semantics, reliable structure, and transferable geometry across extremely different viewpoints.

Problem

• cross-view localization
• spatial intelligence
• semantic consistency
• 3D grounding
• geometric reasoning

Innovation

• cross-view localization
• 3D-grounded alignment
• structure-aware matching
• semantic consistency
• transferable geometry