Foundation Model-Driven Framework for Human-Object Interaction Prediction with Segmentation Mask Integration

2025-04-28

AI Summary

Traditional detection-driven HOI methods struggle to produce pixel-level interaction localization. To address this, we propose Seg2HOI—a novel framework that pioneers the deep integration of promptable vision foundation models (e.g., SAM) into HOI detection, enabling end-to-end generation of HOI quaternaries: (human, object, interaction, segmentation mask). Our method introduces a promptable HOI decoder that jointly models interaction semantics and mask generation, leveraging the zero-shot generalization and multimodal (textual/visual) prompting capabilities of foundation models without additional training. On HICO-DET and VCOCO benchmarks, Seg2HOI achieves state-of-the-art performance; under zero-shot settings, it matches fully supervised methods and reliably produces high-quality HOI segmentation even for unseen prompts. The core contributions are: (1) a new HOI quaternary representation paradigm explicitly incorporating segmentation masks, and (2) the principled, architecture-aware fusion of foundation models with structured HOI reasoning.

Technology Category

• Computer Vision: Segmentation
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

In this work, we introduce Segmentation to Human-Object Interaction ( extit{ extbf{Seg2HOI}}) approach, a novel framework that integrates segmentation-based vision foundation models with the human-object interaction task, distinguished from traditional detection-based Human-Object Interaction (HOI) methods. Our approach enhances HOI detection by not only predicting the standard triplets but also introducing quadruplets, which extend HOI triplets by including segmentation masks for human-object pairs. More specifically, Seg2HOI inherits the properties of the vision foundation model (e.g., promptable and interactive mechanisms) and incorporates a decoder that applies these attributes to HOI task. Despite training only for HOI, without additional training mechanisms for these properties, the framework demonstrates that such features still operate efficiently. Extensive experiments on two public benchmark datasets demonstrate that Seg2HOI achieves performance comparable to state-of-the-art methods, even in zero-shot scenarios. Lastly, we propose that Seg2HOI can generate HOI quadruplets and interactive HOI segmentation from novel text and visual prompts that were not used during training, making it versatile for a wide range of applications by leveraging this flexibility.

Problem

• Integrates segmentation models with human-object interaction prediction
• Extends HOI triplets to quadruplets by adding segmentation masks
• Enables zero-shot HOI detection from novel text and visual prompts

Innovation

• Integrates segmentation masks with HOI prediction
• Uses vision foundation model properties
• Generates HOI quadruplets from novel prompts