Decoding Task Progress from VLA Representations

2026-08-13

AI Summary

This work addresses the lack of interpretability and runtime monitoring in existing vision–language–action (VLA) models. By applying mechanistic interpretability techniques, the authors discover that task-progress signals are linearly decodable from the residual stream of pretrained vision backbones such as PaliGemma—even without any robot data—and generalize robustly to unseen tasks and linguistic counterfactuals. This study is the first to demonstrate that high-level semantic information like task progress is already embedded in pretrained representations. The proposed lightweight linear probe enables real-time detection of task stagnation and achieves out-of-distribution (OOD) detection performance on par with state-of-the-art methods, despite operating without access to labeled data. These findings offer a practical, deployable solution for monitoring VLA systems during execution.

Abstract

Vision-language-action models (VLAs) are moving rapidly towards deployment as general-purpose manipulation policies, but we currently lack basic tools for understanding what these models represent internally or for monitoring them at runtime. Leveraging ideas from mechanistic interpretability, we probe the residual stream of $π_{0.5}$ and find that task progress, the normalized time remaining in a trajectory, is linearly readable from the activations. We find that this signal is present in the pretrained PaliGemma backbone prior to training on any robot-specific data. A single linear probe generalizes to unseen tasks and varies under language counterfactuals when trained on multi-prompt data, but does not enable meaningful steering of the policy. These properties make the signal directly useful for instrumenting deployed VLAs. We use the probe as a simple label-free OOD detector, which detects stalled task progress, and find it competitive with state-of-the-art methods. Our results suggest that VLAs have rich, linearly readable internal representations of semantic quantities like task progress, and that learning to read these signals offers a lightweight, interpretable path toward monitoring deployed visuomotor policies.

Problem

• vision-language-action models
• task progress
• mechanistic interpretability
• runtime monitoring
• internal representations

Innovation

• task progress
• vision-language-action models
• mechanistic interpretability
• linear probing
• out-of-distribution detection