Inference-Time Attention Steering for Vision-Language-Action Driving Models

2026-08-17

AI Summary

研究通过在视觉标记上施加有界加性预softmax注意力偏置，无需重新训练即可引导VLA驾驶模型在推理时关注安全关键行为者。

Abstract

Vision-language-action (VLA) driving models couple a reasoning stage with a diffusion-based trajectory decoder, but do not give a direct way to redirect attention toward safety-critical actors at inference time without retraining. We studied a bounded additive pre-softmax attention bias on the visual tokens of detector localized traffic actors on Alpamayo-R1's Qwen3-VL backbone. It is applied as a fail open forward pre-hook with no weight changes. On 50 lane-change scenarios from the Physical AI World Model Synthetic dataset. The trajectory decoder shows a monotonic dose response in the bias magnitude, separate from a paired zero bias control at every tested magnitude. It reaches $\approx 17$\,cm mean displacement with lateral shifts up to $\sim 140$\ cm at the clamp. A layer ablation places the action-relevant signal in late layers, where the effect increases with the number of hooked layers (2.0cm for the first 8 layers; 67.6cm for all 36). A per call injection audit explains why the Chain-of-Causation text never changes. The mask based bias never reaches the reasoning pathway in this serving stack, so the invariance is verified exposure, not robustness. Steered trajectories tend to shift toward the attended actor, suggesting the bias governs where the model looks rather than encoding a target behavior.

Problem

• inference-time attention
• vision-language-action models
• safety-critical actors
• attention steering
• trajectory decoder

Innovation

• attention steering
• inference-time modification
• bounded additive pre-softmax attention bias
• trajectory decoder