No More Sibling Rivalry: Debiasing Human-Object Interaction Detection

2025-08-31

AI Summary

This paper addresses the degradation of discriminative capability in human-object interaction (HOI) detection caused by mutual interference among highly similar “toxic sibling” triplets at both encoder input and decoder output stages. To mitigate this, we propose a dual-perspective debiasing learning framework: (1) a “contrastive-then-calibration” strategy that integrates strong positional priors into a detection Transformer to guide contrastive learning and calibrate feature representations, thereby alleviating input-side confusion among sibling classes; and (2) a “merge-then-split” strategy that employs hierarchical feature aggregation followed by semantic differentiation to suppress output-side semantic competition and enhance fine-grained interaction discrimination. Evaluated on HICO-Det, our method achieves 38.92% mAP—improving over the baseline by 9.18% and surpassing the prior state-of-the-art by 3.59%. The framework demonstrates consistent and significant gains across multiple evaluation settings.

Technology Category

• Computer Vision: Object Detection & Categorization
• Machine Learning: Calibration & Uncertainty Quantification
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Detection transformers have been applied to human-object interaction (HOI) detection, enhancing the localization and recognition of human-action-object triplets in images. Despite remarkable progress, this study identifies a critical issue-"Toxic Siblings" bias-which hinders the interaction decoder's learning, as numerous similar yet distinct HOI triplets interfere with and even compete against each other both input side and output side to the interaction decoder. This bias arises from high confusion among sibling triplets/categories, where increased similarity paradoxically reduces precision, as one's gain comes at the expense of its toxic sibling's decline. To address this, we propose two novel debiasing learning objectives-"contrastive-then-calibration" and "merge-then-split"-targeting the input and output perspectives, respectively. The former samples sibling-like incorrect HOI triplets and reconstructs them into correct ones, guided by strong positional priors. The latter first learns shared features among sibling categories to distinguish them from other groups, then explicitly refines intra-group differentiation to preserve uniqueness. Experiments show that we significantly outperform both the baseline (+9.18% mAP on HICO-Det) and the state-of-the-art (+3.59% mAP) across various settings.

Problem

• Addressing Toxic Siblings bias in HOI detection
• Reducing confusion among similar interaction triplets
• Improving precision for human-action-object recognition

Innovation

• Contrastive-then-calibration learning for input debiasing
• Merge-then-split objective for output differentiation
• Sibling triplet reconstruction using positional priors