Pi-HOC: Pairwise 3D Human-Object Contact Estimation

2026-04-14

AI Summary

This work addresses the challenge of modeling fine-grained, concurrent 3D physical contact relationships in multi-person, multi-object scenarios. It proposes the first single-pass, instance-aware dense contact prediction framework, which generates dedicated human-object (HO) tokens for each person–object pair and employs an InteractionFormer to model their interactions. A SAM-based decoder predicts semantic contact regions directly on SMPL human meshes without requiring prior knowledge of object geometry. The method innovatively enables multi-person support and zero-shot language-referenced contact prediction. Evaluated on the MMHOI and DAMON datasets, it significantly outperforms existing approaches, achieving simultaneous improvements in localization accuracy and inference throughput—accelerating runtime by 20×—and demonstrates practical utility in optimizing SAM-3D reconstructions.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Computer Vision: 3D Computer Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Resolving real-world human-object interactions in images is a many-to-many challenge, in which disentangling fine-grained concurrent physical contact is particularly difficult. Existing semantic contact estimation methods are either limited to single-human settings or require object geometries (e.g., meshes) in addition to the input image. Current state-of-the-art leverages powerful VLM for category-level semantics but struggles with multi-human scenarios and scales poorly in inference. We introduce Pi-HOC, a single-pass, instance-aware framework for dense 3D semantic contact prediction of all human-object pairs. Pi-HOC detects instances, creates dedicated human-object (HO) tokens for each pair, and refines them using an InteractionFormer. A SAM-based decoder then predicts dense contact on SMPL human meshes for each human-object pair. On the MMHOI and DAMON datasets, Pi-HOC significantly improves accuracy and localization over state-of-the-art methods while achieving 20x higher throughput. We further demonstrate that predicted contacts improve SAM-3D image-to-mesh reconstruction via a test-time optimization algorithm and enable referential contact prediction from language queries without additional training.

Problem

• human-object interaction
• 3D contact estimation
• multi-human scenarios
• instance-aware prediction
• dense semantic contact

Innovation

• 3D human-object contact
• instance-aware modeling
• InteractionFormer
• dense contact prediction
• language-guided contact