Hg-I2P: Bridging Modalities for Generalizable Image-to-Point-Cloud Registration via Heterogeneous Graphs

2026-03-29

AI Summary

This work addresses the challenge of cross-modal feature generalization between images and point clouds, which suffer from significant modality gaps and lead to a notable drop in registration accuracy in unseen scenes. To overcome this, the paper proposes a novel registration method based on heterogeneous graph neural networks. It introduces, for the first time, a heterogeneous graph mechanism that jointly models correspondences between 2D image regions and 3D point cloud regions. Cross-modal features are adaptively aligned through multi-path feature interactions guided by heterogeneous edges, while reliable matches are refined via intra-graph node-edge consistency constraints. Evaluated across six indoor and outdoor cross-domain benchmarks, the proposed method consistently outperforms existing approaches in both registration accuracy and generalization capability.

Abstract

Image-to-point-cloud (I2P) registration aims to align 2D images with 3D point clouds by establishing reliable 2D-3D correspondences. The drastic modality gap between images and point clouds makes it challenging to learn features that are both discriminative and generalizable, leading to severe performance drops in unseen scenarios. We address this challenge by introducing a heterogeneous graph that enables refining both cross-modal features and correspondences within a unified architecture. The proposed graph represents a mapping between segmented 2D and 3D regions, which enhances cross-modal feature interaction and thus improves feature discriminability. In addition, modeling the consistency among vertices and edges within the graph enables pruning of unreliable correspondences. Building on these insights, we propose a heterogeneous graph embedded I2P registration method, termed Hg-I2P. It learns a heterogeneous graph by mining multi-path feature relationships, adapts features under the guidance of heterogeneous edges, and prunes correspondences using graph-based projection consistency. Experiments on six indoor and outdoor benchmarks under cross-domain setups demonstrate that Hg-I2P significantly outperforms existing methods in both generalization and accuracy. Code is released on https://github.com/anpei96/hg-i2p-demo.

Problem

• image-to-point-cloud registration
• modality gap
• generalization
• cross-modal correspondence
• feature discriminability

Innovation

• heterogeneous graph
• image-to-point-cloud registration
• cross-modal feature learning
• correspondence pruning
• generalizable perception