🤖 AI Summary
This work addresses two key challenges in unsupervised keypoint detection: the absence of annotated training data and the lack of cross-view stability guarantees. To this end, we propose NeSS-ST—a novel method that synergistically integrates the Shi-Tomasi handcrafted detector with a Neural Stability Score (NeSS). NeSS is learned self-supervisely by enforcing response consistency of detected keypoints under geometric perturbations and viewpoint variations, thereby guiding the network to learn features that are both accurate and robust across views. The framework further incorporates multi-scale feature fusion and joint regression-classification optimization. Evaluated on HPatches, ScanNet, MegaDepth, and IMC-PT benchmarks, NeSS-ST achieves state-of-the-art performance in keypoint detection, significantly improving generalization for downstream tasks such as feature matching and pose estimation—without requiring manual annotations or dense correspondence supervision.
📝 Abstract
Learning a feature point detector presents a challenge both due to the ambiguity of the definition of a keypoint and, correspondingly, the need for specially prepared ground truth labels for such points. In our work, we address both of these issues by utilizing a combination of a hand-crafted Shi-Tomasi detector, a specially designed metric that assesses the quality of keypoints, the stability score (SS), and a neural network. We build on the principled and localized keypoints provided by the Shi-Tomasi detector and learn the neural network to select good feature points via the stability score. The neural network incorporates the knowledge from the training targets in the form of the neural stability score (NeSS). Therefore, our method is named NeSS-ST since it combines the Shi-Tomasi detector and the properties of the neural stability score. It only requires sets of images for training without dataset pre-labeling or the need for reconstructed correspondence labels. We evaluate NeSS-ST on HPatches, ScanNet, MegaDepth and IMC-PT demonstrating state-of-the-art performance and good generalization on downstream tasks. The project repository is available at: https://github.com/KonstantinPakulev/NeSS-ST.
