Fast Enough to Act: Spatio-Temporal Visual Token Merging for Low-Latency Robotic VLMs and VLAs

2026-06-28

AI Summary

High-resolution video inputs impose excessive inference latency on vision-language models, hindering their applicability to real-time robotic control. To address this challenge, this work proposes ST-Merge, a plug-and-play framework that achieves geometrically consistent cross-frame token fusion without requiring retraining. By constructing 3D spatiotemporal coordinates, ST-Merge employs a multi-queue parallel matching and weighted aggregation mechanism to compress redundant tokens, while introducing dynamic rotary positional encoding to correct fusion-induced spatial biases and preserve perceptual accuracy. Evaluated on Qwen2.5-VL, the method delivers a 2× speedup with only a 1% drop in accuracy. When integrated into the π₀.₅ policy at 1024×1024 resolution, it achieves an 8.3× acceleration while maintaining task success rates on par with the baseline.

Abstract

Vision-language models and vision-language action models endow the robot with unprecedented capabilities. However, the input of video and high-resolution images yields a massive number of visual tokens, leading to extremely high inference latency and severely hindering the robot's real-time control. To break through this computational bottleneck, we propose ST-Merge, a plug-and-play, training-free framework that efficiently fuses redundant tokens directly during the visual encoding phase. By explicitly constructing 3D spatiotemporal coordinates, it employs a multi-queue parallel matching and weighted aggregation mechanism to achieve efficient and geometrically consistent fusion of redundant tokens across frames. In addition, we introduce a post-merge positional correction mechanism that effectively eliminates spatial deviation caused by merging by dynamically re-evaluating the rotational position code of the weighted centroid of the vision token, thereby ensuring the high-precision spatial awareness required for dexterous operation. In the Video Question Answering task on the mainstream VLM, Qwen2.5-VL, ST-Merge achieves a 2$\times$ inference speedup with only a tiny 1\% loss in precision. When deployed on the $π_{0.5}$ VLA policy, ST-Merge achieves an 8.3$\times$ speedup at 1024 $\times$ 1024 resolution and matches the baseline success rate at this high-resolution setting. At lower resolutions, it introduces a small drop in accuracy.

Problem

• vision-language models
• visual tokens
• inference latency
• real-time control
• robotic VLMs

Innovation

• Spatio-Temporal Token Merging
• Low-Latency VLM/VLA
• Training-Free Compression
• Positional Correction
• Real-Time Robotic Vision