GKDT: General Keypoint Detection Transformer

2026-07-01

AI Summary

This work addresses the challenge of keypoint detection in open-domain, multi-task scenarios by introducing MegaKPT, the first unified benchmark encompassing 29 datasets and 1.3 million instances. The authors propose a general-purpose keypoint detection model, GKDT, built upon the DINOv3 Transformer architecture, which supports visual prompts, textual prompts, and multimodal inputs. By integrating multimodal prompt training with a dynamic importance sampling strategy, GKDT achieves significantly enhanced generalization capabilities. Evaluated on 22 test sets, the single-model variant attains over 90% PCK@0.1 accuracy for the majority of categories, demonstrating its exceptional versatility and practical utility in diverse keypoint detection tasks.

Abstract

With the emergence of various pre-trained vision and language models, computer vision is shifting from narrow-domain to open-domain recognition. The construction of a more powerful yet general keypoint detection (GKD) model to support diverse tasks has become increasingly important in the field. To this end, we firstly present a large-scale unified keypoint dataset called MegaKPT. The dataset is composed of over 1.3 million diverse object instances from twenty-nine existing datasets, and enjoys high-quality unified annotations with keypoint text descriptions. Based on MegaKPT, we develop GKDT, a simple, flexible and powerful DINOv3 based Transformer model for General Keypoint Detection. Our GKDT supports visual prompts, text prompts, or both. To enhance model training, we also propose a suite of useful strategies such as mix-modal prompted training and dynamic importance sampling. By testing over 22 test sets with seen or unseen objects, our single GKDT model shows strong performance and generality in detecting keypoints on broad categories, with most categories over 90\% PCK@0.1 accuracy, offering high practical applicability to real-world problems. The dataset, models, and codes will be released at https://github.com/AlanLuSun/General-Keypoint-Detection.

Problem

• General Keypoint Detection
• Open-domain Recognition
• Vision-Language Models
• Keypoint Annotation
• Unified Dataset

Innovation

• General Keypoint Detection
• Vision-Language Model
• Prompt-based Learning
• Unified Keypoint Dataset
• Transformer Architecture