Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs

2026-06-30

AI Summary

This work addresses the challenge of catastrophic forgetting in parameter-constrained multimodal large language models when incorporating tactile perception, which often degrades pre-existing vision–language capabilities. To mitigate this, the authors propose Splash, a framework that quantifies the importance of pretrained parameters and partitions the model into critical and dormant subspaces. The critical subspace is frozen to preserve visual knowledge, while only the dormant subspace is fine-tuned to align tactile and linguistic representations. Splash introduces a novel masking-based isolation mechanism that enables non-intrusive multimodal expansion. Evaluated on tactile–visual benchmarks including SSVTP, TVL, and TacQuad, the method achieves state-of-the-art performance while fully retaining the model’s original general-purpose capabilities and without increasing LLM inference overhead.

Abstract

Touch supplies the physical grounding needed to perceive intrinsic material properties, such as friction and compliance, that vision alone often cannot resolve. Recent efforts for equipping multimodal LLMs with this tactile sense, however, expose a zero-sum trade-off: the limited parameter budget of compact models forces a choice between acquiring the new sensory modality and preserving the established vision-language reasoning. We present Splash, a mask-isolated tactile alignment learning framework for MLLMs. Splash quantifies the significance of each pretrained parameter, and partitions the parameter space into a dormant and critical subspace. While the frozen critical subspace acts as a stable anchor to safeguard general visual knowledge, Splash updates the isolated dormant subspace to internalize tactile alignment towards LLMs. This selective, non-destructive expansion effectively prevents catastrophic forgetting and ensures non-destructive modality expansion. Extensive experiments show that Splash effectively achieves tactile reasoning without additional inference overhead in the LLM part, demonstrating state-of-the-art performance on visuo-tactile benchmarks, including SSVTP, TVL, and TacQuad, while preserving its original general-purpose capabilities.

Problem

• tactile perception
• multimodal LLMs
• catastrophic forgetting
• modality expansion
• parameter efficiency

Innovation

• mask-isolated learning
• tactile alignment
• parameter partitioning
• non-destructive modality expansion
• multimodal LLMs