Look, Zoom, Understand: The Robotic Eyeball for Embodied Perception

2025-11-19

AI Summary

Existing embodied AI perception systems are constrained by fixed RGB-D cameras or static vision models, limiting their ability to simultaneously achieve wide-area coverage and fine-grained observation. This paper introduces EyeVLA—a command-driven robotic active eye system that enables high-precision visual foveation on targets within large-scale scenes via controllable actions such as rotation and zooming. Our key contributions are threefold: (1) the first integration of open-world understanding capabilities from vision-language models (VLMs) into active visual policy learning; (2) a novel action tokenization scheme for discrete control and a 2D bounding-box coordinate-guided reasoning chain for spatial grounding; and (3) reinforcement learning–based optimization of viewpoint selection. Evaluated in real-world settings with minimal training data, EyeVLA significantly improves command-following accuracy and fine-grained observational fidelity, delivering high-quality visual inputs for downstream embodied tasks.

Technology Category

• Computer Vision: Language and Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

In embodied AI perception systems, visual perception should be active: the goal is not to passively process static images, but to actively acquire more informative data within pixel and spatial budget constraints. Existing vision models and fixed RGB-D camera systems fundamentally fail to reconcile wide-area coverage with fine-grained detail acquisition, severely limiting their efficacy in open-world robotic applications. To address this issue, we propose EyeVLA, a robotic eyeball for active visual perception that can take proactive actions based on instructions, enabling clear observation of fine-grained target objects and detailed information across a wide spatial extent. EyeVLA discretizes action behaviors into action tokens and integrates them with vision-language models (VLMs) that possess strong open-world understanding capabilities, enabling joint modeling of vision, language, and actions within a single autoregressive sequence. By using the 2D bounding box coordinates to guide the reasoning chain and applying reinforcement learning to refine the viewpoint selection policy, we transfer the open-world scene understanding capability of the VLM to a vision language action (VLA) policy using only minimal real-world data. Experiments show that our system efficiently performs instructed scenes in real-world environments and actively acquires more accurate visual information through instruction-driven actions of rotation and zoom, thereby achieving strong environmental perception capabilities. EyeVLA introduces a novel robotic vision system that leverages detailed and spatially rich, large-scale embodied data, and actively acquires highly informative visual observations for downstream embodied tasks.

Problem

• Reconcile wide-area coverage with fine-grained detail acquisition in robotics
• Enable active visual perception through instruction-driven robotic actions
• Transfer open-world scene understanding to vision-language-action policies efficiently

Innovation

• Active robotic eyeball with rotation and zoom
• Integrates action tokens with vision-language models
• Reinforcement learning refines viewpoint selection policy