Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models

2026-06-17

AI Summary

This work addresses the unclear extent to which fine-tuned vision-language-action (VLA) models retain commonsense and world knowledge, a challenge compounded by the difficulty of disentangling knowledge deficits from poor low-level control generalization. To this end, we propose Act2Answer, a novel lightweight evaluation protocol that treats object-placement actions as answer outputs, thereby assessing knowledge retention while minimizing control-related confounds. We construct a tabletop action-grounded question-answering benchmark with a multi-category knowledge test set and employ hierarchical intent probes to analyze information distribution between the vision-language model (VLM) backbone and the action head. Experiments across seven VLA models and nine VLM baselines reveal that VLAs perform well on simple concepts but exhibit significant knowledge degradation on complex semantic categories; joint VQA training aids knowledge preservation; and answer-relevant signals peak in intermediate layers, attenuating in higher ones.

Abstract

Embodied Vision-Language-Action (VLA) models are typically obtained by fine-tuning powerful pretrained VLMs on robotics data, yet it is unclear how much commonsense and factual knowledge they retain after adaptation. Failures on knowledge-sensitive tasks are ambiguous, conflating missing knowledge with poor generalization of low-level control. We introduce Act2Answer, a lightweight protocol that adapts VLM knowledge benchmarks to VLA evaluation by requiring agents to answer through action. Each question becomes a short tabletop episode where the agent performs a single object-placement action to select among candidate answers, yielding an action-grounded success rate with reduced control confounds. We curate a test suite of such environments across diverse commonsense and world-knowledge categories and introduce layerwise intent probing to localize answer-relevant information across the VLM backbone and action head. In a large-scale study of 7 VLA models and 9 VLM baselines, we systematically rank models across categories, finding that VLAs show solid performance on simple concepts while exhibiting larger gaps on richer semantic categories relative to their source VLMs, that VQA co-training is associated with better knowledge retention, and that answer-relevant signals peak in middle VLA layers but attenuate in upper layers. Act2Answer is available at https://tttonyalpha.github.io/act2answer/.

Problem

• Vision-Language-Action models
• commonsense knowledge
• world knowledge retention
• knowledge evaluation
• embodied AI

Innovation

• Act2Answer
• Vision-Language-Action models
• commonsense knowledge retention
• action-grounded evaluation
• layerwise intent probing