Diff-Reg v2: Diffusion-Based Matching Matrix Estimation for Image Matching and 3D Registration

2025-03-06

AI Summary

This work addresses correspondence ambiguity in 2D image, 3D point cloud, and 2D–3D cross-modal registration—arising from scale disparity, geometric symmetry, and large non-rigid deformations. We propose the first diffusion-based framework operating directly in the matching matrix space. Our method formulates matching matrix estimation as an iterative denoising process within the Birkhoff polytope of doubly stochastic matrices, incorporating task-adaptive matrix embeddings and a unified “match-to-warp” encoding paradigm. To enforce row- and column-wise stochastic constraints, we introduce dual Softmax projection regularization. Departing from single-step prediction, our lightweight denoising module drives multi-step reverse sampling, significantly enhancing robustness. Experiments demonstrate substantial improvements in matching accuracy across diverse registration benchmarks, with particularly pronounced gains under severe deformations, symmetric structures, and cross-scale scenarios.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Matrix & Tensor Methods
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Security and Privacy: Large-scale security measurements
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Establishing reliable correspondences is crucial for all registration tasks, including 2D image registration, 3D point cloud registration, and 2D-3D image-to-point cloud registration. However, these tasks are often complicated by challenges such as scale inconsistencies, symmetry, and large deformations, which can lead to ambiguous matches. Previous feature-based and correspondence-based methods typically rely on geometric or semantic features to generate or polish initial potential correspondences. Some methods typically leverage specific geometric priors, such as topological preservation, to devise diverse and innovative strategies tailored to a given enhancement goal, which cannot be exhaustively enumerated. Additionally, many previous approaches rely on a single-step prediction head, which can struggle with local minima in complex matching scenarios. To address these challenges, we introduce an innovative paradigm that leverages a diffusion model in matrix space for robust matching matrix estimation. Our model treats correspondence estimation as a denoising diffusion process in the matching matrix space, gradually refining the intermediate matching matrix to the optimal one. Specifically, we apply the diffusion model in the doubly stochastic matrix space for 3D-3D and 2D-3D registration tasks. In the 2D image registration task, we deploy the diffusion model in a matrix subspace where dual-softmax projection regularization is applied. For all three registration tasks, we provide adaptive matching matrix embedding implementations tailored to the specific characteristics of each task while maintaining a consistent"match-to-warp"encoding pattern. Furthermore, we adopt a lightweight design for the denoising module. In inference, once points or image features are extracted and fixed, this module performs multi-step denoising predictions through reverse sampling.

Problem

• Addresses challenges in image and 3D registration tasks.
• Overcomes issues like scale inconsistencies and large deformations.
• Introduces diffusion-based matching matrix estimation for robust correspondences.

Innovation

• Uses diffusion model for robust matching matrix estimation
• Applies diffusion in doubly stochastic matrix space
• Implements adaptive matching matrix embedding