EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training

2026-04-21

AI Summary

Current vision-language-action models (VLAs) directly employ off-the-shelf vision-language models (VLMs) that are not tailored to embodied settings, leading to suboptimal performance. This work proposes EmbodiedMidtrain, which for the first time systematically characterizes the distributional discrepancy between VLMs and VLAs and introduces a learnable proximity estimator to select task-aligned samples from diverse VLM pretraining data during an intermediate training phase, achieving alignment at both sample and dataset levels. By doing so, it enhances spatial reasoning capabilities while preserving the semantic richness of the original VLM. Experiments demonstrate that EmbodiedMidtrain significantly improves performance across three robotic manipulation benchmarks, matching the efficacy of expert-level VLAs. The approach is compatible with various VLM backbones and exhibits consistent gains early in training, maintaining its advantage throughout.

Technology Category

• Intelligent Robots: Embodied AI
• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Vision-Language-Action Models (VLAs) inherit their visual and linguistic capabilities from Vision-Language Models (VLMs), yet most VLAs are built from off-the-shelf VLMs that are not adapted to the embodied domain, limiting their downstream performance. In this work, we propose EmbodiedMidtrain to bridge the gap between VLMs and VLAs. We first characterize the data distribution gap between them, showing that VLA data occupy compact regions that are largely separated from the broader VLM distribution, while the degree of alignment varies substantially both across and within VLM data sources. Then, we build a mid-training data engine that leverages a lightweight learnable proximity estimator to select the most VLA-aligned candidates from a large VLM pool, and mid-trains the VLM on this curated mixture before downstream VLA fine-tuning. Experiments on three robot manipulation benchmarks show that mid-training consistently improves performance across different VLM backbones, achieving results competitive with expert VLAs and off-the-shelf VLMs trained with larger model scale and training budgets. Further analysis reveals that mid-training provides a stronger initialization for VLA fine-tuning, with gains emerging from the earliest steps and widening throughout training. Moreover, the data engine captures both dataset-level and sample-level alignment signals, favoring spatial reasoning over text-centric tasks while preserving the diversity of the VLM data. We will release all code, data and models for future research.

Problem

• Vision-Language Models
• Vision-Language-Action Models
• embodied AI
• data distribution gap
• mid-training

Innovation

• EmbodiedMidtrain
• Vision-Language-Action Models
• mid-training
• data alignment
• proximity estimator