SAFE-Pruner: Semantic Attention-Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulation

2026-05-28

AI Summary

Existing visual-language-action (VLA) models often prune visual tokens based on shallow features, which risks discarding critical deep-layer information and consequently degrades inference stability and task success rates. To address this, this work proposes SAFE-Pruner, a plug-and-play pruning framework that, for the first time, leverages cross-layer attention consistency of the same semantic entity within VLA models to design a forward-looking token saliency prediction mechanism. It further incorporates an adaptive subtask partitioning strategy to detect attention shifts, enabling precise identification of important tokens. Experiments demonstrate that SAFE-Pruner achieves up to 1.89× inference acceleration in both simulated and real-world settings, with less than a 1.7% drop in task success rate, outperforming the current state-of-the-art method by 1.9%.

Abstract

Real-time inference of vision-language-action (VLA) models is essential for robotic control. While visual token pruning has shown strong potential for accelerating inference, most existing methods mainly base pruning decisions on shallow-layer cues and risk discarding visual information required by deep layers. To address this issue, we propose SAFE-Pruner, a plug-and-play pruning framework that incorporates attention cues of future layers into pruning decisions. Specifically, we identify semantic attention consistency, the tendency that VLA models concentrate their attention probability mass on the same semantic entity across execution steps. Based on this observation, we design a forward-looking strategy to forecast the token saliency in deep layers, which prevents the premature removal of critical tokens and leads to more stable acceleration. We further introduce an adaptive subtask division strategy to detect abrupt attention shifts, thereby improving forecasting accuracy and pruning reliability. Extensive experiments in simulation and real-world settings demonstrate that our method achieves up to 1.89x speedup with a minimal degradation in success rate of less than 1.7%, while outperforming state-of-the-art methods by up to 1.9%.

Problem

• vision-language-action
• token pruning
• real-time inference
• semantic attention
• deep layers

Innovation

• semantic attention consistency
• future-aware pruning
• token saliency forecasting
• adaptive subtask division
• vision-language-action models