CLAP: Direct VLM-to-VLA Adaptation via Language-Action Grounding

2026-07-09

AI Summary

This work addresses the semantic degradation often observed when directly transferring pretrained vision-language models (VLMs) to vision-language-action (VLA) models, primarily caused by output distribution mismatches. To mitigate this, the authors propose CLAP, a method that prefixes natural language action descriptions before action tokens, thereby causally conditioning action prediction on language-action plans. This approach enables efficient and transparent transfer from multi-scale VLMs to VLAs with only a single round of fine-tuning and without altering the backbone architecture. A family of VLA models (0.8B/2B/4B parameters) built upon a unified VLM backbone preserves original language capabilities while significantly enhancing action generation robustness. Notably, the 2B variant achieves a 90.8% success rate on the LIBERO benchmark—14.9 percentage points higher than VLA-0—and demonstrates superior performance under various perturbations in LIBERO-PRO.

Abstract

Vision-language-action models (VLAs) inherit semantic capabilities from pretrained VLMs, yet large-scale post-training on robot data and architectural modifications can reshape the backbone so extensively that it becomes difficult to isolate what the VLM contributes to control. Directly converting pretrained VLMs into VLAs with minimal architectural change offers a more transparent path to understanding how VLM capabilities transfer across model scales. The core obstacle is output-distribution mismatch: predicting actions as bare numeric token sequences moves generation away from the VLM's pretrained language distribution, degrading the capabilities we seek to preserve. To address this, we propose CLAP (Causal Language-Action Prediction), which prepends each numeric action sequence with a natural-language action description, causally conditioning precise action-token prediction on a language-action plan without modifying the backbone architecture. With single-epoch fine-tuning alone, 2B CLAP achieves 90.8% on LIBERO (+14.9 pt over VLA-0) and improves robustness on LIBERO-PRO under language, object, and spatial perturbations. We will release CLAP at 0.8B, 2B, and 4B as an open-weight, multi-scale compact VLA family from a single VLM lineage, enabling controlled analysis of VLM-to-VLA capability transfer.

Problem

• vision-language-action models
• output-distribution mismatch
• VLM-to-VLA adaptation
• language-action grounding
• capability transfer

Innovation

• CLAP
• vision-language-action models
• language-action grounding
• output-distribution alignment
• minimal architectural adaptation