RaCo: Ranking and Covariance for Practical Learned Keypoints

2026-02-17

AI Summary

This work addresses the challenge of learning highly repeatable, rotation-robust keypoints with metric-scale uncertainty without relying on co-visible image pairs. The authors propose RaCo, a lightweight neural network that, for the first time, enables end-to-end joint learning of keypoint repeatability ranking and metric covariance estimation using only single-view images and without any additional supervision. RaCo integrates a lightweight convolutional architecture, a differentiable ranking mechanism, a covariance estimation module, and large-scale perspective-cropping augmentation, thereby avoiding costly equivariant structures. Experimental results demonstrate that RaCo achieves state-of-the-art performance in keypoint repeatability and two-view matching across multiple challenging benchmarks, with particularly strong robustness under large in-plane rotations.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: State Estimation
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

This paper introduces RaCo, a lightweight neural network designed to learn robust and versatile keypoints suitable for a variety of 3D computer vision tasks. The model integrates three key components: the repeatable keypoint detector, a differentiable ranker to maximize matches with a limited number of keypoints, and a covariance estimator to quantify spatial uncertainty in metric scale. Trained on perspective image crops only, RaCo operates without the need for covisible image pairs. It achieves strong rotational robustness through extensive data augmentation, even without the use of computationally expensive equivariant network architectures. The method is evaluated on several challenging datasets, where it demonstrates state-of-the-art performance in keypoint repeatability and two-view matching, particularly under large in-plane rotations. Ultimately, RaCo provides an effective and simple strategy to independently estimate keypoint ranking and metric covariance without additional labels, detecting interpretable and repeatable interest points. The code is available at https://github.com/cvg/RaCo.

Problem

• learned keypoints
• keypoint repeatability
• spatial uncertainty
• rotational robustness
• two-view matching

Innovation

• learned keypoints
• differentiable ranking
• covariance estimation
• rotation robustness
• single-view training