Premover: Fast Vision-Language-Action Control by Acting Before Instructions Are Complete

2026-05-12

AI Summary

This work addresses the significant interaction latency in conventional vision-language-action (VLA) systems, which wait until user instructions are fully received before execution. To mitigate this delay, the authors propose Premover, a lightweight module that leverages idle time during streaming instruction input to perform safe pre-actions before command completion. Premover operates on a frozen VLA backbone augmented with a small projection head, generating image-language attention maps in a shared embedding space and dynamically determining execution timing via a learnable readiness threshold. The attention maps are trained under simulation-based target mask supervision, while the readiness threshold is jointly optimized. Evaluated on the LIBERO benchmark, Premover reduces average interaction time from 34.0 to 29.4 seconds (a 13.6% improvement) while maintaining a high success rate of 95.1%, substantially outperforming naive early-execution baselines (66.4%).

Abstract

Vision-Language-Action (VLA) policies are typically evaluated as if the user had finished typing or speaking before the robot begins acting. In real deployment, however, users take several seconds to enter a request, leaving the policy idle for a substantial fraction of the interaction. We introduce Premover, a lightweight module that converts this idle window into useful precomputation. Premover keeps the VLA backbone frozen and attaches two small projection heads, one for image patches, one for language tokens, that map an intermediate layer of the backbone into a shared space. The resulting focus map is supervised by simulator-rendered target-object segmentation masks and applied as a per-patch reweighting of the next step's image tokens. A single scalar readiness threshold, trained jointly from streaming prefixes, decides when the policy should begin acting. On the LIBERO benchmark suite, Premover reduces mean wall-clock time from 34.0 to 29.4 seconds, a 13.6% reduction, while matching the full-prompt baseline's success rate (95.1% vs. 95.0%); naive premoving, by contrast, collapses to 66.4%.

Problem

• Vision-Language-Action
• premature action
• interactive latency
• instruction delay
• real-time control

Innovation

• Premover
• Vision-Language-Action
• precomputation
• streaming instructions
• focus map