Single-Query Person-Centric Bimanual Hand-Object Interaction Detection

2026-09-10

AI Summary

本文提出了一种以人为核心的单查询方法，通过部分感知的可变形注意力机制解决双手与物体互动的识别问题，统一了检测、姿态估计和手部推理。

Abstract

Understanding person-level bi-manual interactions requires not only detecting hands, but also identifying which two hands belong to the same person and what each hand interacts with. Existing hand--object interaction methods are mostly hand-centric: they treat each hand as an independent instance, which can lead to ambiguous ownership in multi-person scenes. We propose a person-centric formulation in which a single query predicts a structured output for one person, including the human box, body pose, hand boxes and states, and interaction targets. We introduce part-aware deformable attention to allocate attention across human, hand, and pose-specific reference regions, enabling one query to capture the full person structure. We further unify detection and interaction reasoning with a hand-to-query relationship matrix, where each hand selects its interaction target from the detected query set plus a learnable off token, directly recovering the target's box and class without separate object regression. We build a COCO-based dataset with person-centric bi-manual interaction annotations and define structured metrics for evaluating hand states and complete hand--object tuples. Experiments with a transformer-based detector show that our formulation improves person-level bi-manual interaction parsing and provides an effective unified framework for joint detection, pose estimation, and hand reasoning.

Problem

• bi-manual interaction
• hand-centric methods
• person-centric
• interaction ambiguity
• multi-person scenes

Innovation

• person-centric
• part-aware deformable attention
• hand-to-query relationship matrix