MatchAnything: Universal Cross-Modality Image Matching with Large-Scale Pre-Training

2025-01-13

AI Summary

Cross-modal image matching suffers from poor generalization due to large appearance discrepancies and scarce annotated data—especially in scientific imaging. To address this, we propose the first large-scale cross-modal pretraining framework based on synthetic signals. Our method generates controllable cross-modal registration signals, integrates heterogeneous multi-source data, and introduces a structure-aware feature alignment module alongside a transfer-enhanced matching head to learn universal structural correspondences across modalities. Crucially, a single pretrained model achieves zero-shot transfer to over eight unseen cross-modal tasks—including MRI/CT, visible/infrared—and consistently outperforms state-of-the-art methods. It attains top or near-top performance across multiple real-world benchmarks, breaking the conventional modality-specific training paradigm. This work establishes a new low-resource paradigm for cross-modal registration, enabling robust, generalizable matching without task-specific fine-tuning.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

Image matching, which aims to identify corresponding pixel locations between images, is crucial in a wide range of scientific disciplines, aiding in image registration, fusion, and analysis. In recent years, deep learning-based image matching algorithms have dramatically outperformed humans in rapidly and accurately finding large amounts of correspondences. However, when dealing with images captured under different imaging modalities that result in significant appearance changes, the performance of these algorithms often deteriorates due to the scarcity of annotated cross-modal training data. This limitation hinders applications in various fields that rely on multiple image modalities to obtain complementary information. To address this challenge, we propose a large-scale pre-training framework that utilizes synthetic cross-modal training signals, incorporating diverse data from various sources, to train models to recognize and match fundamental structures across images. This capability is transferable to real-world, unseen cross-modality image matching tasks. Our key finding is that the matching model trained with our framework achieves remarkable generalizability across more than eight unseen cross-modality registration tasks using the same network weight, substantially outperforming existing methods, whether designed for generalization or tailored for specific tasks. This advancement significantly enhances the applicability of image matching technologies across various scientific disciplines and paves the way for new applications in multi-modality human and artificial intelligence analysis and beyond.

Problem

• Image Matching
• Cross-Modality
• Deep Learning

Innovation

• MatchAnything
• Cross-type Image Matching
• Large-scale Learning