Doodle Your Keypoints: Sketch-Based Few-Shot Keypoint Detection

2025-07-10

AI Summary

Few-shot keypoint detection suffers from the absence of in-distribution source data for supervision. Method: This paper introduces, for the first time, user-drawn sketches as an unsupervised, source-free supervisory signal. To bridge the cross-modal gap between sketches and images and mitigate individual drawing-style bias, we propose a prototype-based framework integrating a grid-based localizer and a prototype-level domain adaptation mechanism, enabling sketch–image semantic alignment and precise keypoint regression. Contribution/Results: Our method requires no pre-trained source-domain data and supports rapid generalization to novel categories and unseen keypoints. Extensive experiments on multiple benchmarks demonstrate that it significantly outperforms existing few-shot keypoint detection approaches, achieving strong robustness across diverse sketch styles and high localization accuracy.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Keypoint detection, integral to modern machine perception, faces challenges in few-shot learning, particularly when source data from the same distribution as the query is unavailable. This gap is addressed by leveraging sketches, a popular form of human expression, providing a source-free alternative. However, challenges arise in mastering cross-modal embeddings and handling user-specific sketch styles. Our proposed framework overcomes these hurdles with a prototypical setup, combined with a grid-based locator and prototypical domain adaptation. We also demonstrate success in few-shot convergence across novel keypoints and classes through extensive experiments.

Problem

• Addresses few-shot keypoint detection without same-distribution source data
• Leverages sketches for cross-modal embedding and style challenges
• Proposes prototypical framework for novel keypoints and classes

Innovation

• Leveraging sketches for source-free keypoint detection
• Prototypical setup with grid-based locator
• Prototypical domain adaptation for cross-modal embeddings