JamMa: Ultra-lightweight Local Feature Matching with Joint Mamba

2025-03-05

AI Summary

Existing Transformer-based feature matchers effectively model long-range dependencies but suffer from high spatial complexity, prohibitive training costs, and substantial inference latency—hindering the simultaneous achievement of accuracy and efficiency. To address this, we propose the first ultra-lightweight Mamba-based matcher for local feature matching, introducing two novel designs: Joint Mamba modeling and JEGO (Joint Efficient Global-Order) scanning. These enable dual-image joint high-frequency interaction, stride-efficient sequential scanning, global receptive fields, and omnidirectional feature representation. By incorporating state-space modeling, sparse sequence modeling, and bidirectional feature coupling, our architecture reduces parameter count and FLOPs to under 50% of mainstream attention-based matchers. It converges on a single GPU and achieves significantly faster inference while attaining state-of-the-art accuracy–efficiency trade-off.

Technology Category

• Computer Vision: Multi-modal Vision
• Search and Optimization: Learning to Search
• Machine Learning: Mixture of Experts (MoE)

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Existing state-of-the-art feature matchers capture long-range dependencies with Transformers but are hindered by high spatial complexity, leading to demanding training and highlatency inference. Striking a better balance between performance and efficiency remains a challenge in feature matching. Inspired by the linear complexity O(N) of Mamba, we propose an ultra-lightweight Mamba-based matcher, named JamMa, which converges on a single GPU and achieves an impressive performance-efficiency balance in inference. To unlock the potential of Mamba for feature matching, we propose Joint Mamba with a scan-merge strategy named JEGO, which enables: (1) Joint scan of two images to achieve high-frequency mutual interaction, (2) Efficient scan with skip steps to reduce sequence length, (3) Global receptive field, and (4) Omnidirectional feature representation. With the above properties, the JEGO strategy significantly outperforms the scan-merge strategies proposed in VMamba and EVMamba in the feature matching task. Compared to attention-based sparse and semi-dense matchers, JamMa demonstrates a superior balance between performance and efficiency, delivering better performance with less than 50% of the parameters and FLOPs.

Problem

• Balancing performance and efficiency in feature matching
• Reducing spatial complexity in feature matchers
• Achieving high-frequency mutual interaction in image scans

Innovation

• Mamba-based matcher with linear complexity
• Joint Mamba with JEGO scan-merge strategy
• Efficient feature matching with reduced parameters