Action QFormer: Structured Representation Shaping under Action Supervision in Vision-Language-Action Models

2026-07-16

AI Summary

This work addresses a critical limitation in vision-language-action models: directly applying action supervision to multimodal representation pathways degrades language understanding and object grounding while hindering the construction of structured action representations. To overcome this, the authors propose Action QFormer, which uniquely treats action supervision as a structural signal for shaping multimodal representations. By introducing an instruction-conditioned query mechanism, Action QFormer selectively reorganizes information from pretrained vision-language features to generate action-oriented structured representations without compromising upstream linguistic capabilities. This design effectively decouples action learning from interference with language comprehension. Experiments demonstrate substantial improvements—closed-loop success rate in zero-shot sim-to-real navigation rises from 18.8% to 56.3%, fixed-instruction action generation accuracy increases from 22.5% to 75.5%, and out-of-distribution instruction following is significantly suppressed.

Abstract

Action supervision in vision-language-action (VLA) models is often treated as a downstream objective for learning action prediction. In this paper, we study it instead as a force that shapes inherited multimodal representations. We show that this shaping has a dual effect: it is necessary for forming action-compatible representations, but when action supervision is applied too directly to the inherited multimodal pathway, it can also destabilize representations that support language-side processing and object grounding. To address this tension, we introduce Action QFormer, a query-based action-facing interface that uses instruction-conditioned queries to reorganize inherited multimodal information into action-facing representations before downstream action generation. In zero-shot sim-to-real navigation, Action QFormer improves average closed-loop task success from 18.8% to 56.3%, raises fixed-instruction action-generation correctness from 22.5% to 75.5%, and nearly eliminates out-of-distribution instruction generations. Further analyses show that Action QFormer changes how action supervision shapes inherited multimodal representations, reducing broad upstream rewriting while preserving targeted and sometimes constructive action-supervised adaptation. These results suggest that improving VLA performance requires not only stronger pretrained backbones, but also better ways of selecting and organizing inherited multimodal information while controlling how it is shaped under action supervision.

Problem

• vision-language-action models
• action supervision
• multimodal representations
• representation shaping
• instruction-conditioned queries

Innovation

• Action QFormer
• vision-language-action models
• action supervision
• structured representation
• query-based interface