Modular Sensory Stream for Integrating Physical Feedback in Vision-Language-Action Models

2026-04-25

AI Summary

Existing vision-language-action models struggle to effectively integrate heterogeneous physical feedback signals, limiting their perception and decision-making capabilities in real-world settings. This work proposes MoSS, a framework that employs decoupled, modular sensory streams to flexibly incorporate multimodal physical signals such as tactile and torque data, and introduces a joint cross-modal self-attention mechanism to enhance modeling of contact dynamics. By combining a two-stage training strategy with an auxiliary task of future signal prediction, MoSS achieves stable and efficient fusion of multiple physical modalities. Real-robot experiments demonstrate that the proposed approach significantly improves action prediction performance, validating the efficacy and added value of synergistically integrating multimodal physical feedback.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Humans understand and interact with the real world by relying on diverse physical feedback beyond visual perception. Motivated by this, recent approaches attempt to incorporate physical sensory signals into Vision-Language-Action models (VLAs). However, they typically focus on a single type of physical signal, failing to capture the heterogeneous and complementary nature of real-world interactions. In this paper, we propose MoSS, a modular sensory stream framework that adapts VLAs to leverage multiple sensory signals for action prediction. Specifically, we introduce decoupled modality streams that integrate heterogeneous physical signals into the action stream via joint cross-modal self-attention. To enable stable incorporation of new modalities, we adopt a two-stage training scheme that freezes pretrained VLA parameters in the early stage. Furthermore, to better capture contact interaction dynamics, we incorporate an auxiliary task that predicts future physical signals. Through extensive real-world experiments, we demonstrate that MoSS successfully augments VLAs to leverage diverse physical signals (i.e., tactile and torque), integrating multiple signals to achieve synergistic performance gains.

Problem

• Vision-Language-Action models
• physical feedback
• heterogeneous sensory signals
• multimodal integration
• action prediction

Innovation

• modular sensory stream
• vision-language-action models
• heterogeneous physical signals
• cross-modal self-attention
• multi-sensory integration