Image-to-Point Cloud Registration Made Easy with Rectified Flow-based LiDAR Upsampling

2026-07-16

AI Summary

This work addresses the challenges of low registration accuracy and poor generalization arising from the modality gap between images and point clouds by treating LiDAR as an imaging sensor and proposing a self-supervised registration framework. The method first generates dense intensity images from sparse LiDAR scans using a conditional Rectified Flow model, then performs cross-modal feature matching with camera images via a pre-trained matcher, and finally estimates the six-degree-of-freedom relative pose using PnP-RANSAC. Requiring neither image–point cloud correspondences nor ground-truth pose labels, the approach achieves strong generalization with only minimal fine-tuning on LiDAR data. Evaluated on the R3LIVE dataset, it attains an average pose error of 4.89°/1.63 m with a runtime of approximately 0.68 seconds per registration, significantly outperforming existing methods.

Abstract

Image-to-Point Cloud Registration (I2P) is essential for integrating camera and LiDAR in perception and autonomous systems, yet the modality gap between images and point clouds makes it difficult to achieve both high accuracy and strong generalization. In this paper, we propose a simple yet effective I2P method that treats LiDAR as an imaging sensor: from a single sparse LiDAR scan, we generate a dense LiDAR intensity image using Conditional Rectified Flow, match it with a camera image using a pre-trained feature matcher, and estimate the 6-DoF relative pose via PnP-RANSAC. The proposed model is pre-trained through a self-supervised image completion task and fine-tuned on a small amount of LiDAR data (neither image-point cloud pairs nor ground-truth sensor poses are required), enabling it to scale to diverse LiDAR and camera configurations. Experiments on the R3LIVE dataset show that the proposed method achieves a mean error of 4.89° / 1.63 m, outperforming existing methods, while completing a single registration in approximately 0.68 s.

Problem

• Image-to-Point Cloud Registration
• modality gap
• LiDAR
• camera
• 6-DoF pose estimation

Innovation

• Image-to-Point Cloud Registration
• Rectified Flow
• LiDAR Upsampling
• Self-supervised Learning
• 6-DoF Pose Estimation