Incremental Object Keypoint Learning

2025-03-26

AI Summary

Existing object keypoint estimation models suffer from poor generalization to novel keypoints introduced during testing, limiting their adaptability to downstream tasks. To address this, we propose Incremental Keypoint Learning (IKL), a paradigm enabling continual model expansion using only annotations for new keypoints—without replaying old data. Our method introduces a two-stage knowledge association framework: (i) a Knowledge Association Network (KA-Net) jointly encodes spatial and anatomical constraints, and (ii) a keypoint-guided spatial distillation loss that facilitates knowledge transfer from old to new keypoints. Notably, IKL achieves forward transfer—the training of new keypoints improves the accuracy of previously learned ones—a first in keypoint estimation. Experiments demonstrate substantial mitigation of catastrophic forgetting, strong performance under low-shot annotation budgets, and consistent superiority over state-of-the-art methods across multiple benchmarks.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Computer Vision: Large Vision Models
• Knowledge Representation and Reasoning: Knowledge Acquisition

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Existing progress in object keypoint estimation primarily benefits from the conventional supervised learning paradigm based on numerous data labeled with pre-defined keypoints. However, these well-trained models can hardly detect the undefined new keypoints in test time, which largely hinders their feasibility for diverse downstream tasks. To handle this, various solutions are explored but still suffer from either limited generalizability or transferability. Therefore, in this paper, we explore a novel keypoint learning paradigm in that we only annotate new keypoints in the new data and incrementally train the model, without retaining any old data, called Incremental object Keypoint Learning (IKL). A two-stage learning scheme as a novel baseline tailored to IKL is developed. In the first Knowledge Association stage, given the data labeled with only new keypoints, an auxiliary KA-Net is trained to automatically associate the old keypoints to these new ones based on their spatial and intrinsic anatomical relations. In the second Mutual Promotion stage, based on a keypoint-oriented spatial distillation loss, we jointly leverage the auxiliary KA-Net and the old model for knowledge consolidation to mutually promote the estimation of all old and new keypoints. Owing to the investigation of the correlations between new and old keypoints, our proposed method can not just effectively mitigate the catastrophic forgetting of old keypoints, but may even further improve the estimation of the old ones and achieve a positive transfer beyond anti-forgetting. Such an observation has been solidly verified by extensive experiments on different keypoint datasets, where our method exhibits superiority in alleviating the forgetting issue and boosting performance while enjoying labeling efficiency even under the low-shot data regime.

Problem

• Detect undefined new keypoints in test time
• Overcome limited generalizability and transferability
• Mitigate catastrophic forgetting of old keypoints

Innovation

• Incremental keypoint learning without old data
• Two-stage learning with Knowledge Association
• Mutual Promotion via spatial distillation loss