Incremental Human-Object Interaction Detection with Invariant Relation Representation Learning

2025-10-30

AI Summary

Open-world human-object interaction (HOI) detection faces persistent challenges—including continual evolution of interactions, interaction drift across tasks, and frequent zero-shot compositional generalization—rendering conventional incremental learning approaches inadequate. Method: This paper proposes the first exemplar-free incremental HOI detection framework. Its core innovation is an incremental relation distillation mechanism that decouples object detection from relational reasoning and introduces two dedicated distillation losses to learn task-invariant relational feature representations, thereby jointly mitigating catastrophic forgetting and interaction drift without accessing historical samples or exemplars. Results: The method achieves state-of-the-art performance on HICO-DET and V-COCO, significantly outperforming existing approaches. It demonstrates superior zero-shot generalization capability and enhanced stability in continual learning settings, establishing a new benchmark for exemplar-free incremental HOI detection.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Reasoning under Uncertainty: Relational Probabilistic Models
• Computer Vision: Object Detection & Categorization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

In open-world environments, human-object interactions (HOIs) evolve continuously, challenging conventional closed-world HOI detection models. Inspired by humans' ability to progressively acquire knowledge, we explore incremental HOI detection (IHOID) to develop agents capable of discerning human-object relations in such dynamic environments. This setup confronts not only the common issue of catastrophic forgetting in incremental learning but also distinct challenges posed by interaction drift and detecting zero-shot HOI combinations with sequentially arriving data. Therefore, we propose a novel exemplar-free incremental relation distillation (IRD) framework. IRD decouples the learning of objects and relations, and introduces two unique distillation losses for learning invariant relation features across different HOI combinations that share the same relation. Extensive experiments on HICO-DET and V-COCO datasets demonstrate the superiority of our method over state-of-the-art baselines in mitigating forgetting, strengthening robustness against interaction drift, and generalization on zero-shot HOIs. Code is available at href{https://github.com/weiyana/ContinualHOI}{this HTTP URL}

Problem

• Detects evolving human-object interactions in dynamic environments
• Addresses catastrophic forgetting and interaction drift in incremental learning
• Enables generalization to zero-shot HOI combinations without exemplars

Innovation

• Decouples object and relation learning
• Uses invariant relation distillation losses
• Learns features across shared relations