TAIHRI: Task-Aware 3D Human Keypoints Localization for Close-Range Human-Robot Interaction

2026-04-10

AI Summary

Existing 3D human keypoint estimation methods prioritize full-body reconstruction, which fails to meet the demand for metrically accurate localization of task-relevant body parts in the robot’s camera coordinate system during close-proximity human-robot interaction. This work proposes TAIHRI, the first vision-language model tailored for this scenario, which interprets natural language action instructions to focus on task-critical regions and quantizes 3D keypoints into a bounded interaction space. By integrating 2D keypoint reasoning with next-token prediction, TAIHRI achieves precise 3D localization. It pioneers the use of vision-language models in proximal human-robot interaction perception, unifying multimodal inputs and spatial outputs, and significantly outperforms existing approaches on egocentric interaction benchmarks in localizing task-relevant body parts. The framework further supports downstream applications such as natural-language-controlled interaction and human mesh recovery.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Human-Robot Interaction
• Computer Vision: Language and Vision

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Accurate 3D human keypoints localization is a critical technology enabling robots to achieve natural and safe physical interaction with users. Conventional 3D human keypoints estimation methods primarily focus on the whole-body reconstruction quality relative to the root joint. However, in practical human-robot interaction (HRI) scenarios, robots are more concerned with the precise metric-scale spatial localization of task-relevant body parts under the egocentric camera 3D coordinate. We propose TAIHRI, the first Vision-Language Model (VLM) tailored for close-range HRI perception, capable of understanding users'motion commands and directing the robot's attention to the most task-relevant keypoints. By quantizing 3D keypoints into a finite interaction space, TAIHRI precisely localize the 3D spatial coordinates of critical body parts by 2D keypoint reasoning via next token prediction, and seamlessly adapt to downstream tasks such as natural language control or global space human mesh recovery. Experiments on egocentric interaction benchmarks demonstrate that TAIHRI achieves superior estimation accuracy for task-critical body parts. We believe TAIHRI opens new research avenues in the field of embodied human-robot interaction. Code is available at: https://github.com/Tencent/TAIHRI.

Problem

• 3D human keypoints localization
• human-robot interaction
• task-aware perception
• egocentric vision
• metric-scale accuracy

Innovation

• Vision-Language Model
• 3D Human Keypoints Localization
• Task-Aware Perception
• Egocentric Interaction
• Human-Robot Interaction