Attention from Action, for Action: Emergent Visual Bottlenecks for Policy Learning

2026-08-13

AI Summary

This work addresses the limitations of existing visual policy learning methods, which rely on fixed or manually annotated regions of interest and thus struggle to adapt to dynamically shifting visual attention during tasks, resulting in poor data efficiency and robustness. The authors propose Seeker, a self-supervised approach that learns task-aware attention without external annotations by leveraging action signals alone. Built upon frozen DINOv2 features, Seeker iteratively generates dynamic regions of interest conditioned on both state and task, which guide image cropping, background augmentation, and point cloud filtering. Evaluated in both simulated and real-world robotic settings, the method substantially improves performance: task success rates on physical robots increase from 48.3% to 76.7%, and under lighting and background perturbations, from 20.0% to 60.0%.

Abstract

Visual bottlenecks that focus policy inputs on regions of interest (ROIs) can improve data-efficient visuomotor learning by separating where to look from how to act. Many ROI interfaces rely on external spatial labels, such as gaze, object classes, or affordance annotations. Label-free alternatives often derive crops from trajectories by detecting gripper or motion events and centering a fixed crop at the projected end-effector. Such action-derived crops are useful spatial priors that require no additional labels, but they encode fixed choices about event timing, proxy points, and crop scale. When the visual evidence needed for control lies away from the end-effector or changes continuously with task progress, these crops can become misaligned. We propose Seeker, a task- and state-conditioned readout that learns attention from action. Starting from frozen DINOv3 features, Seeker iteratively updates a query with gathered visual evidence, producing progression-aware ROIs solely from action supervision. The learned ROI serves as a spatial interface for RGB cropping, mask-guided background augmentation, and point-cloud filtering. In simulation and the real world, Seeker improves data efficiency and robustness over no-crop, augmentation, and action-derived crop baselines. On real robots, Seeker raises average in-domain success from the best baseline's 48.3% to 76.7% and success under lighting/background shifts from 20.0% to 60.0%.

Problem

• visual bottlenecks
• region of interest
• visuomotor learning
• action-derived crops
• spatial priors

Innovation

• attention from action
• visual bottleneck
• region of interest
• policy learning
• label-free attention