Improving and Evaluating Hand-Object Interaction Detection

2026-06-15

AI Summary

This work addresses the challenge of detecting hand–object interactions, including those mediated by tools, by proposing HOI-DETR—a novel framework that, for the first time, jointly models hand–object and object–object interactions within a Co-DETR architecture while integrating multimodal features and temporal context. The authors establish a unified evaluation benchmark encompassing Hands23, HOIST, FineBio, and HD-EPIC, offering high-quality annotations and video-level evaluation protocols. The proposed method achieves substantial performance gains, improving mean average precision (mAP) by over 20 percentage points on both Hands23 and FineBio, and sets new state-of-the-art results across all four datasets, thereby advancing standardization and pushing the performance boundaries in the field of hand–object interaction understanding.

Abstract

Understanding hands and the objects they interact with, both directly and through tools, is a key step for tasks ranging from action perception to 3D reconstruction and robotics. Our paper provides several contributions to the Hand-Object Interaction (HOI) understanding literature: (1) HOI-DETR, a new framework that introduces hand-object and object-object interactions to the Co-DETR architecture to produce a state-of-the-art method; (2) a comprehensive HOI evaluation suite of 4 diverse datasets, including a video benchmark derived from the HD-EPIC dataset and fresh annotations that improve the Hands23 benchmark and (3) a trained checkpoint that significantly improves the state of the art across Hands23, HOIST, FineBio, and HD-EPIC, including mAP gains of over 20 percentage points on Hands23 and FineBio. Our ablations confirm the contributions of each model component.

Problem

• Hand-Object Interaction
• Interaction Detection
• HOI Evaluation
• 3D Reconstruction
• Action Perception

Innovation

• HOI-DETR
• hand-object interaction
• object-object interaction
• comprehensive evaluation suite
• state-of-the-art detection