ForgeVLA: Federated Vision-Language-Action Learning without Language Annotations

2026-05-08

AI Summary

This work addresses the challenges of training vision-language-action (VLA) models, which typically rely on costly language annotations and struggle to leverage distributed, heterogeneous robotic data. To overcome these limitations, the authors propose a federated learning framework that operates without centralized raw data or manual language labels. Each client employs an embodied instruction classifier to map vision-action pairs onto a predefined set of instructions, thereby reconstructing the missing language modality and forming complete vision-language-action triplets. The approach jointly optimizes multimodal representations through a client-side contrastive planning loss and a server-side adaptive aggregation strategy, effectively mitigating feature collapse. This method represents the first successful realization of language-label-free federated VLA training, significantly outperforming existing approaches across multiple benchmarks, with ablation studies confirming the contribution of each component.

Abstract

Vision-Language-Action (VLA) models hold great promise for general-purpose robotic intelligence, yet scaling up such models is severely bottlenecked by the high cost of acquiring annotated training data. Fortunately, vision-equipped robots deployed across various domains already produce abundant vision-action pairs that can be leveraged to scale up VLA training more efficiently. However, these raw data cannot be centrally aggregated due to various constraints and also exhibit severe heterogeneity. To address these challenges, in this paper, we propose ForgeVLA, a federated VLA training framework that learns VLA models from distributed vision-action pairs without centralizing raw data or requiring manual annotations. Specifically, each client in ForgeVLA is equipped with an embodied instruction classifier that maps vision-action pairs to a predefined instruction set, recovering the missing language modality and forming complete vision-language-action triplets. Beyond triplet construction, we also identify vision-language feature collapse as a critical challenge that has been largely overlooked in prior federated VLA research. To mitigate this issue, ForgeVLA combines a client-side contrastive planning loss with a server-side adaptive aggregation strategy to learn task-discriminative representations efficiently. Extensive experiments across multiple benchmarks show that ForgeVLA significantly outperforms other baselines, and ablation studies further validate the contribution of each component.

Problem

• Federated Learning
• Vision-Language-Action Models
• Language Annotations
• Data Heterogeneity
• Robotic Intelligence

Innovation

• Federated Learning
• Vision-Language-Action Models
• Embodied Instruction Classification
• Feature Collapse Mitigation
• Contrastive Planning Loss