AnyMatch: Supercharging Universal Multi-Modal Image Matching with Large-Scale Single-View Images

2026-06-29

AI Summary

Existing methods are hindered by the scarcity of large-scale, geometrically accurate, and photorealistic multimodal image pairs; real-world datasets are costly and limited in scene diversity, while synthetic data often fails to simultaneously achieve geometric consistency and visual realism. This work proposes a novel paradigm that dispenses with conventional SfM-MVS pipelines and instead leverages only single-view images to automatically generate geometrically consistent, visually realistic multimodal multi-view image pairs along with precise annotations. The approach integrates monocular depth estimation, 3D reprojection, diffusion-based inpainting, and cross-modal image translation, enabling controllable scene diversity and annotation difficulty. Matching models (e.g., LoFTR, EDM, RoMa) fine-tuned on the resulting Any-syn dataset significantly outperform current state-of-the-art methods on multimodal benchmarks, demonstrating superior generalization and robustness.

Abstract

Multi-modal image matching is essential for visual localization and multi-sensor fusion, but it is hindered by the scarcity of large-scale training data with precise geometric annotations. Existing real-world datasets suffer from prohibitive costs, limited scene diversity, and errors in SfM-MVS pipelines, while synthetic methods struggle to maintain 3D geometric consistency or achieve photorealistic appearance. To address this, we propose AnyMatch, a novel framework that leverages abundant, easily accessible single-view images at minimal cost to generate rich multi-modal training data. AnyMatch integrates monocular depth estimation, 3D reprojection, diffusion-based inpainting, and crossmodal image translation to synthesize multi-view, multi-modal image pairs with 3D geometric fidelity. Crucially, our method provides annotations that strictly adhere to 3D geometric consistency through explicit 3D reprojection, avoiding SfM-MVS error accumulation. Furthermore, AnyMatch offers strong scalability, enabling controllable scene diversity and annotation difficulty via adjustable input and camera parameters. We construct Any-syn, a large-scale synthetic multi-modal dataset using AnyMatch. Experimental results show that matching networks (e.g., LoFTR, EDM, RoMa) fine-tuned on Any-syn achieve substantial performance gains on multi-modal benchmarks, exhibiting superior generalization and robustness compared to models trained on existing data.

Problem

• multi-modal image matching
• training data scarcity
• 3D geometric consistency
• photorealistic appearance
• visual localization

Innovation

• multi-modal image matching
• 3D geometric consistency
• single-view image synthesis
• diffusion-based inpainting
• scalable synthetic dataset