GazeVLA: Learning Human Intention for Robotic Manipulation

2026-04-24

AI Summary

This work addresses the limited generalization of robotic manipulation methods that rely heavily on large-scale demonstration data by proposing a cross-modal transfer framework that leverages human gaze as an observable proxy for task intent. The approach first pretrains a vision-language-action (VLA) model via self-supervision on large-scale egocentric human videos, then fine-tunes it with minimal paired robot and human data. During inference, a chain-of-thought mechanism is introduced to first predict high-level task intent—explicitly modeled through human gaze as an intermediate representational bridge—and subsequently generate actions. This explicit modeling of gaze effectively mitigates embodiment discrepancies between humans and robots. Experiments demonstrate that the method significantly outperforms strong baselines across both simulation and real-world settings, achieving state-of-the-art performance on long-horizon, fine-grained tasks under few-shot and robustness evaluation protocols.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Embodied foundation models have achieved significant breakthroughs in robotic manipulation, yet they still depend heavily on large-scale robot demonstrations. Although recent works have explored leveraging human data to alleviate this dependency, effectively extracting transferable knowledge remains a significant challenge due to the inherent embodiment gap between human and robot. We argue that the intention underlying human actions can serve as a powerful intermediate representation for bridging this gap. In this paper, we introduce a novel framework that explicitly learns and transfers human intention to facilitate robotic manipulation. Specifically, we model intention through gaze, as it naturally precedes physical actions and serves as an observable proxy for human intent. Our model is first pretrained on a large-scale egocentric human dataset to capture human intention and its synergy with action, followed by finetuning on a small set of robot and human data. During inference, the model adopts a Chain-of-Thought reasoning paradigm, sequentially predicting intention before executing the action. Extensive evaluations in simulation and real-world settings, across long-horizon and fine-grained tasks, and under few-shot and robustness benchmarks, show that our method consistently outperforms strong baselines, generalizes better, and achieves state-of-the-art performance.

Problem

• embodiment gap
• human intention
• robotic manipulation
• transferable knowledge
• human-robot transfer

Innovation

• human intention
• gaze modeling
• embodiment gap
• Chain-of-Thought reasoning
• embodied foundation models