MinCD-PnP: Learning 2D-3D Correspondences with Approximate Blind PnP

2025-07-21

AI Summary

In image-to-point-cloud (I2P) registration, learning 2D–3D correspondences is highly susceptible to noise and outliers, while conventional differentiable PnP solvers are overly sensitive to incorrect matches. To address this, we propose MinCD-PnP, an approximate blind PnP method that formulates registration as minimizing the Chamfer distance between 2D detections and 3D keypoints—bypassing explicit correspondence assignment and thereby enhancing robustness and computational efficiency. This work is the first to integrate the blind PnP paradigm into an end-to-end learnable registration framework. We design MinCD-Net, a lightweight multi-task network jointly optimizing keypoint regression, Chamfer distance minimization, and differentiable PnP. Evaluated on 7-Scenes, RGB-D-V2, and ScanNet, MinCD-Net achieves state-of-the-art performance, with significant improvements in inlier ratio and registration recall. Moreover, it demonstrates strong cross-scene and cross-dataset generalization capability.

Technology Category

• Computer Vision: Learning & Optimization for CV
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Image-to-point-cloud (I2P) registration is a fundamental problem in computer vision, focusing on establishing 2D-3D correspondences between an image and a point cloud. The differential perspective-n-point (PnP) has been widely used to supervise I2P registration networks by enforcing the projective constraints on 2D-3D correspondences. However, differential PnP is highly sensitive to noise and outliers in the predicted correspondences. This issue hinders the effectiveness of correspondence learning. Inspired by the robustness of blind PnP against noise and outliers in correspondences, we propose an approximated blind PnP based correspondence learning approach. To mitigate the high computational cost of blind PnP, we simplify blind PnP to an amenable task of minimizing Chamfer distance between learned 2D and 3D keypoints, called MinCD-PnP. To effectively solve MinCD-PnP, we design a lightweight multi-task learning module, named as MinCD-Net, which can be easily integrated into the existing I2P registration architectures. Extensive experiments on 7-Scenes, RGBD-V2, ScanNet, and self-collected datasets demonstrate that MinCD-Net outperforms state-of-the-art methods and achieves a higher inlier ratio (IR) and registration recall (RR) in both cross-scene and cross-dataset settings.

Problem

• Improving robustness in 2D-3D correspondence learning against noise
• Reducing computational cost of blind PnP for efficient registration
• Enhancing cross-scene and cross-dataset registration accuracy

Innovation

• Approximated blind PnP for robust correspondence learning
• Minimizing Chamfer distance between 2D and 3D keypoints
• Lightweight multi-task learning module (MinCD-Net)