A Study of Failure Modes in Two-Stage Human-Object Interaction Detection

2026-04-15

AI Summary

Current two-stage human-object interaction (HOI) detection models exhibit insufficient robustness in complex scenarios—such as those involving multiple interacting agents or rare actions—and their overall accuracy often masks fine-grained failure modes. This work proposes a fine-grained analysis framework that does not require constructing new large-scale benchmarks. By partitioning datasets according to scene configuration categories, the framework dissects model behavior across multiple dimensions and attributes specific failure patterns. The approach systematically uncovers fundamental limitations in the visual reasoning capabilities of existing models, revealing significant performance degradation in challenging configurations like object sharing and dense interactions. These findings demonstrate that conventional aggregate metrics inadequately reflect true reasoning competence, thereby offering an interpretable pathway for diagnosing and improving HOI models.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Human-object interaction (HOI) detection aims to detect interactions between humans and objects in images. While recent advances have improved performance on existing benchmarks, their evaluations mainly focus on overall prediction accuracy and provide limited insight into the underlying causes of model failures. In particular, modern models often struggle in complex scenes involving multiple people and rare interaction combinations. In this work, we present a study to better understand the failure modes of two-stage HOI models, which form the basis of many current HOI detection approaches. Rather than constructing a large-scale benchmark, we instead decompose HOI detection into multiple interpretable perspectives and analyze model behavior across these dimensions to study different types of failure patterns. We curate a subset of images from an existing HOI dataset organized by human-object-interaction configurations (e.g., multi-person interactions and object sharing), and analyze model behavior under these configurations to examine different failure modes. This design allows us to analyze how these HOI models behave under different scene compositions and why their predictions fail. Importantly, high overall benchmark performance does not necessarily reflect robust visual reasoning about human-object relationships. We hope that this study can provide useful insights into the limitations of HOI models and offer observations for future research in this area.

Problem

• human-object interaction
• failure modes
• two-stage detection
• complex scenes
• model robustness

Innovation

• failure mode analysis
• two-stage HOI detection
• interpretable evaluation
• human-object interaction
• scene composition