Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild

2026-07-15

AI Summary

Traditional human-object interaction (HOI) detection methods rely on predefined categories and supervised training, limiting their generalization to open-world and compositional scenarios. This work proposes AgentHOI, a novel framework that achieves open-vocabulary HOI detection without any HOI-specific training data. Leveraging multimodal large language models, AgentHOI employs a context-aware, multi-turn reasoning mechanism in synergy with modular vision foundation models to generate instance-level descriptions that integrate semantic, spatial, and appearance cues. A multi-dimensional interaction localization strategy further enables precise discovery of interactions. Evaluated in real-world open settings, AgentHOI substantially outperforms existing supervised and weakly supervised approaches, demonstrating exceptional generalization capability and practical utility.

Abstract

Human-object interaction detection (HOID) has traditionally been formulated as a supervised detection problem over predefined interaction categories. While such paradigms achieve strong performance on closed-set benchmarks, they fundamentally entangle interaction understanding with dataset-specific supervision, limiting their ability to generalize to open-world and compositional scenarios. Recent HOI detectors attempt to leverage MLLMs through prompting strategies to transfer interaction-specific knowledge. However, such prompt-based approaches primarily focus on extracting discriminative representations from pretrained models, while underexploring their inherent multimodal reasoning capabilities. As a result, they struggle to provide informative contextual reasoning for ambiguous and open-world interaction scenarios. In this work, we present AgentHOI, a training-free, agentic framework that transfers the generalist multimodal reasoning capabilities of foundation models to HOI detection in the wild. Instead of learning interaction classifiers, AgentHOI modularly orchestrates complementary vision foundation modules to perform open-ended semantic reasoning and spatial grounding in a coordinated manner. To address the challenges of incomplete interaction discovery and ambiguous localization in complex scenes, we introduce two key mechanisms: (1) Context-aware Multi-round Reasoning, which progressively refines interaction hypotheses to ensure exhaustive and compositional HOI discovery, and (2) Multifaceted Interaction Localization, which enhances grounding precision by generating instance-specific descriptions that integrate semantic, spatial, and appearance cues. Extensive experiments demonstrate that AgentHOI achieves superior performance over state-of-the-art supervised and weakly supervised methods in real-world settings, despite requiring no HOID data for training.

Problem

• Human-Object Interaction Detection
• Open-world Generalization
• Multimodal Reasoning
• Compositional Scenarios
• Training-free

Innovation

• training-free
• multimodal reasoning
• human-object interaction detection
• open-world generalization
• agent-based framework