Recurrent Feature Mining and Keypoint Mixup Padding for Category-Agnostic Pose Estimation

2025-03-27

AI Summary

This work addresses category-agnostic pose estimation—specifically, pixel-level keypoint localization for arbitrary novel categories using only a few annotated support images. To this end, we propose the Fine-Grained Structural Awareness (FGSA) feature mining framework. Our method introduces a deformable-attention-based FGSA module and a multi-scale feature interaction mechanism; incorporates a support-query feature recursive optimization strategy; and pioneers keypoint chain-wise offset modeling alongside Mixup-based padding (replacing zero-padding) to unify the number of keypoints across categories and strengthen supervision signals. Evaluated on the MP-100 benchmark, our approach achieves a 3.2% absolute improvement in PCK@0.05 over prior state-of-the-art methods. The source code is publicly available.

Technology Category

• Computer Vision: Biometrics, Face, Gesture & Pose
• Machine Learning: Mixture of Experts (MoE)
• Intelligent Robots: State Estimation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Category-agnostic pose estimation aims to locate keypoints on query images according to a few annotated support images for arbitrary novel classes. Existing methods generally extract support features via heatmap pooling, and obtain interacted features from support and query via cross-attention. Hence, these works neglect to mine fine-grained and structure-aware (FGSA) features from both support and query images, which are crucial for pixel-level keypoint localization. To this end, we propose a novel yet concise framework, which recurrently mines FGSA features from both support and query images. Specifically, we design a FGSA mining module based on deformable attention mechanism. On the one hand, we mine fine-grained features by applying deformable attention head over multi-scale feature maps. On the other hand, we mine structure-aware features by offsetting the reference points of keypoints to their linked keypoints. By means of above module, we recurrently mine FGSA features from support and query images, and thus obtain better support features and query estimations. In addition, we propose to use mixup keypoints to pad various classes to a unified keypoint number, which could provide richer supervision than the zero padding used in existing works. We conduct extensive experiments and in-depth studies on large-scale MP-100 dataset, and outperform SOTA method dramatically (+3.2%PCK@0.05). Code is avaiable at https://github.com/chenbys/FMMP.

Problem

• Mining fine-grained and structure-aware features for pose estimation
• Improving keypoint localization via recurrent feature mining
• Enhancing supervision with mixup keypoints instead of zero padding

Innovation

• Recurrent mining of FGSA features using deformable attention
• Multi-scale deformable attention for fine-grained features
• Mixup keypoints padding for richer supervision