LiftFeat: 3D Geometry-Aware Local Feature Matching

2025-05-06

AI Summary

To address poor local feature matching robustness in SLAM and visual localization under challenging conditions—such as drastic illumination changes, low-texture regions, and repetitive patterns—this paper proposes LiftFeat, a lightweight neural network. Methodologically, LiftFeat introduces monocular depth–predicted pseudo surface normals as a 3D geometric supervision signal into a lightweight feature matching framework for the first time, and designs a 3D geometry-aware feature enhancement module to effectively fuse geometric priors with 2D descriptors. The method operates solely on monocular RGB input, requiring no additional sensors or dense annotations. Evaluated on relative pose estimation, homography estimation, and visual localization tasks, LiftFeat consistently outperforms state-of-the-art lightweight methods: it achieves up to a 12.7% improvement in matching recall under extreme conditions and a 9.3% gain in localization accuracy.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Feature Construction/Reformulation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Robust and efficient local feature matching plays a crucial role in applications such as SLAM and visual localization for robotics. Despite great progress, it is still very challenging to extract robust and discriminative visual features in scenarios with drastic lighting changes, low texture areas, or repetitive patterns. In this paper, we propose a new lightweight network called extit{LiftFeat}, which lifts the robustness of raw descriptor by aggregating 3D geometric feature. Specifically, we first adopt a pre-trained monocular depth estimation model to generate pseudo surface normal label, supervising the extraction of 3D geometric feature in terms of predicted surface normal. We then design a 3D geometry-aware feature lifting module to fuse surface normal feature with raw 2D descriptor feature. Integrating such 3D geometric feature enhances the discriminative ability of 2D feature description in extreme conditions. Extensive experimental results on relative pose estimation, homography estimation, and visual localization tasks, demonstrate that our LiftFeat outperforms some lightweight state-of-the-art methods. Code will be released at : https://github.com/lyp-deeplearning/LiftFeat.

Problem

• Enhances feature matching in challenging lighting and texture conditions
• Integrates 3D geometry to improve 2D feature discriminability
• Outperforms lightweight methods in pose and homography estimation

Innovation

• Uses monocular depth estimation for surface normal
• Fuses 3D geometric features with 2D descriptors
• Enhances feature robustness in extreme conditions