Semantic Anchoring for Robotic Action Representations

2026-07-15

AI Summary

Fine-tuning vision-language-action (VLA) models often disrupts pretrained semantic structures, degrading both in-distribution task performance and out-of-distribution generalization. Inspired by mirror neuron theory, this work proposes a plug-and-play semantic anchoring mechanism that decomposes action representations into shared and private channels via semantic probing and anchors them to the semantic manifold of the pretrained vision-language model. This approach preserves semantic structure without altering the inference architecture. Extensive experiments across diverse VLA backbones and real-world as well as simulated environments demonstrate its effectiveness, achieving up to an 18.7% improvement in in-distribution task performance and a 21.5% gain in out-of-distribution generalization.

Abstract

Vision-Language-Action (VLA) models inherit rich semantic representations from pretrained Vision-Language Models, yet fine-tuning on limited robot demonstrations degrades this structure and undermines generalization. A fundamental question therefore arises: what constitutes a good action representation? Inspired by the mirror neuron theory's insight that observation and execution share an intention-level encoding, we examine whether a robot's action representations preserve the semantic structure captured by pretrained encoders. Systematic probing confirms that this structure erodes during finetuning, and that its quality synchronizes with both task success and out-of-distribution generalization. We further introduce a plug-and-play method that anchors action representations to a semantic manifold while decomposing representations into a shared semantic channel and a private channel, all discarded at inference, leaving the deployed model unchanged. Validated on different VLA backbones across simulation and real-world benchmarks, our method yields up to +18.7% on real-world in-distribution tasks and +21.5% on out-of-distribution generalization.

Problem

• action representation
• semantic structure
• Vision-Language-Action models
• generalization
• fine-tuning

Innovation

• Semantic Anchoring
• Vision-Language-Action Models
• Action Representation
• Out-of-Distribution Generalization
• Mirror Neuron Theory