From Pairs to Sequences: Track-Aware Policy Gradients for Keypoint Detection

2026-02-24

AI Summary

Existing keypoint detection methods struggle to maintain trackability across long image sequences under varying viewpoints and illumination conditions. This work addresses this limitation by formulating keypoint detection as a sequential decision-making problem and proposes TraqPoint, an end-to-end reinforcement learning framework that directly optimizes tracking quality (Traq) over image sequences. The core innovation lies in a tracking-aware reward mechanism that jointly enforces multi-view geometric consistency and feature discriminability, enabling end-to-end training via policy gradient methods. Experimental results demonstrate that TraqPoint significantly outperforms state-of-the-art keypoint detection and description approaches on sparse matching tasks, including relative pose estimation and 3D reconstruction.

Technology Category

• Computer Vision: Motion & Tracking
• Reasoning under Uncertainty: Sequential Decision Making
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Security and Privacy: Tracking, profiling, and countermeasures against them

Abstract

Keypoint-based matching is a fundamental component of modern 3D vision systems, such as Structure-from-Motion (SfM) and SLAM. Most existing learning-based methods are trained on image pairs, a paradigm that fails to explicitly optimize for the long-term trackability of keypoints across sequences under challenging viewpoint and illumination changes. In this paper, we reframe keypoint detection as a sequential decision-making problem. We introduce TraqPoint, a novel, end-to-end Reinforcement Learning (RL) framework designed to optimize the \textbf{Tra}ck-\textbf{q}uality (Traq) of keypoints directly on image sequences. Our core innovation is a track-aware reward mechanism that jointly encourages the consistency and distinctiveness of keypoints across multiple views, guided by a policy gradient method. Extensive evaluations on sparse matching benchmarks, including relative pose estimation and 3D reconstruction, demonstrate that TraqPoint significantly outperforms some state-of-the-art (SOTA) keypoint detection and description methods.

Problem

• keypoint detection
• trackability
• image sequences
• viewpoint changes
• illumination changes

Innovation

• track-aware reward
• sequential decision-making
• keypoint detection
• reinforcement learning
• trajectory quality