MuseVLA: An Adaptive Multimodal Sensing Vision-Language-Action Model for Robotic Manipulation

2026-06-16

AI Summary

This work addresses the limitations of existing vision-language-action (VLA) models, which predominantly rely on RGB images and thus lack access to critical physical properties such as temperature, sound, or radar signals, hindering their performance in complex manipulation tasks. To overcome this, the authors propose an adaptive multimodal VLA framework that introduces a novel on-demand querying mechanism for non-visual sensors. Heterogeneous sensor inputs are unified into “grounded sensor images” to drive action generation. By decoupling sensor processing from the main backbone and integrating multimodal fusion, sensor token generation, and synthetic data training, the model achieves cross-modal zero-shot generalization. Real-world robotic experiments demonstrate an average success rate of 80.6%, substantially outperforming both RGB-only and other multisensory baselines, while exhibiting strong generalization to unseen tasks.

Abstract

Humans naturally leverage diverse sensing modalities to interact with the physical world, while most Vision-Language-Action (VLA) models for robotics rely solely on RGB observations. This limits their ability to perceive physical properties that are difficult or impossible to infer from RGB cameras, such as temperature, sound, or radar response. We present MuseVLA, an adaptive multimodal sensing VLA model that integrates novel sensors as on-demand tools for robotic manipulation. Given a task instruction and visual context, MuseVLA first generates a sensor token and target description that select the sensing modality to invoke and what to attend to, analogous to a tool call with arguments. It then converts the selected sensor measurement into a grounded sensor image, a unified intermediate representation that encodes heterogeneous readings for multimodal fusion and action generation. This design decouples sensor-specific processing from the VLA backbone, enabling efficient integration of diverse modalities. To reduce the need for expensive multisensory robot datasets, we further introduce a data synthesis pipeline that augments existing RGB video datasets with grounded sensor images, enabling generalization to unseen sensor-guided tasks. We evaluate MuseVLA on a real-world robot across challenging dexterous hand manipulation tasks that require multimodal sensing inputs, including temperature-guided pick-and-place, audio-driven object search, and radar-assisted hidden object retrieval. MuseVLA achieves 80.6% success rate on average, outperforming RGB-only and multisensory VLA baselines significantly, and exhibits strong zero-shot capabilities on unseen tasks.

Problem

• Vision-Language-Action
• multimodal sensing
• robotic manipulation
• physical properties perception
• sensor integration

Innovation

• multimodal sensing
• vision-language-action model
• grounded sensor image
• adaptive sensor selection
• data synthesis