TouchFormer: A Robust Transformer-based Framework for Multimodal Material Perception

2025-11-23

AI Summary

To address the significant degradation in material perception performance under visually constrained conditions, this paper proposes a Transformer-based robust multimodal fusion framework. Unlike naive modality concatenation, our approach introduces a modality-adaptive gating mechanism and cross-instance embedding regularization to dynamically weight modal contributions, explicitly suppress modality-specific noise, and gracefully handle partial modality absence. Additionally, we design both intra- and inter-modality attention modules to enable fine-grained cross-modal feature alignment and complementarity. Evaluated on the SSMC and USMC benchmarks, the framework achieves absolute accuracy improvements of 2.48% and 6.83%, respectively. Furthermore, comprehensive experiments on a real-world robotic platform demonstrate its robustness and generalization capability in complex, visually degraded environments.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Traditional vision-based material perception methods often experience substantial performance degradation under visually impaired conditions, thereby motivating the shift toward non-visual multimodal material perception. Despite this, existing approaches frequently perform naive fusion of multimodal inputs, overlooking key challenges such as modality-specific noise, missing modalities common in real-world scenarios, and the dynamically varying importance of each modality depending on the task. These limitations lead to suboptimal performance across several benchmark tasks. In this paper, we propose a robust multimodal fusion framework, TouchFormer. Specifically, we employ a Modality-Adaptive Gating (MAG) mechanism and intra- and inter-modality attention mechanisms to adaptively integrate cross-modal features, enhancing model robustness. Additionally, we introduce a Cross-Instance Embedding Regularization(CER) strategy, which significantly improves classification accuracy in fine-grained subcategory material recognition tasks. Experimental results demonstrate that, compared to existing non-visual methods, the proposed TouchFormer framework achieves classification accuracy improvements of 2.48% and 6.83% on SSMC and USMC tasks, respectively. Furthermore, real-world robotic experiments validate TouchFormer's effectiveness in enabling robots to better perceive and interpret their environment, paving the way for its deployment in safety-critical applications such as emergency response and industrial automation. The code and datasets will be open-source, and the videos are available in the supplementary materials.

Problem

• Addresses performance degradation in vision-based material perception under impaired conditions
• Solves naive multimodal fusion ignoring noise, missing data, and dynamic importance
• Improves robustness for material recognition in real-world robotic applications

Innovation

• Modality-Adaptive Gating mechanism for cross-modal integration
• Intra- and inter-modality attention for robust fusion
• Cross-Instance Embedding Regularization improves fine-grained recognition