TeaMatch: Teachable Cross-Modal Representation Learning for 2D-3D Matching

2026-08-10

AI Summary

This work addresses the challenge of establishing robust 2D–3D correspondences under severe degradation conditions such as noise, low overlap, and structural ambiguity. The authors propose TeaMatch, a novel framework that introduces teachability into cross-modal representation learning for the first time. By employing a task-oriented weak student to simulate typical failure modes and optimizing representations to recover the teacher’s features, TeaMatch enhances structural consistency and robustness. The approach integrates a teacher–student architecture with correspondence-level constraints and geometry-aware regularization, seamlessly fitting into coarse-to-fine matching pipelines without incurring additional inference overhead. Extensive experiments demonstrate that TeaMatch achieves state-of-the-art performance across multiple challenging 2D–3D matching benchmarks, significantly improving matching robustness.

Abstract

Learning reliable correspondences between images and point clouds is fundamental for 2D-3D matching. Despite recent progress in detection-free methods, existing approaches primarily optimize matching within a single model and often struggle to maintain reliable correspondences under challenging conditions such as noisy inputs, low overlap, and ambiguous structures. In this work, we propose TeaMatch, a novel framework that introduces teachability as a criterion for cross-modal representation learning. We define teachability as the ability of a representation to be effectively recovered by weak learners under degraded inputs, reflecting its structural consistency and robustness. To this end, we construct a set of task-specific weak students that simulate common failure modes and train them to imitate the teacher on a training split while evaluating their recoverability on a disjoint meta split. The teacher is then optimized to improve the students' ability to recover reliable correspondences, guided by correspondence-level and geometry-aware constraints. Our framework can be seamlessly integrated into existing coarse-to-fine matching pipelines without additional inference cost. Extensive experiments demonstrate that TeaMatch improves matching robustness and achieves state-of-the-art performance on challenging 2D-3D matching benchmarks.

Problem

• 2D-3D matching
• cross-modal representation
• correspondence learning
• robustness
• noisy inputs

Innovation

• teachability
• cross-modal representation learning
• 2D-3D matching
• weak learners
• geometry-aware constraints