Heterogeneous Tactile Transformer

2026-06-29

AI Summary

This work addresses the challenge that heterogeneous tactile sensors hinder cross-device transferability, thereby limiting scalable learning of contact-rich manipulation policies. To overcome this, the authors propose the Heterogeneous Tactile Transformer (HTT) framework, which integrates sensor-specific encoders with a shared Transformer backbone to learn a unified tactile representation through intra-modal masked reconstruction and cross-modal alignment. The study introduces HPT, the first large-scale dataset comprising 1.6 million synchronized frames from four distinct tactile sensor modalities, and leverages both visual and tactile array data for multimodal pretraining. Experimental results demonstrate that HTT significantly enhances transfer performance across unseen tactile sensors and novel tasks in diverse perception and real-world manipulation settings.

Abstract

Tactile sensors are inherently heterogeneous: a model trained on one sensor cannot be directly used on another, which limits learning contact-rich manipulation policies from diverse tactile data at scale. To bridge this gap, we propose the Heterogeneous Tactile Transformer (HTT), a framework that learns shared tactile representations across heterogeneous sensors. HTT consists of sensor-specific encoders and a shared transformer trunk, and is pretrained with per-modality masked reconstruction together with cross-modal alignment between paired sensors. Pretraining uses our novel Heterogeneous Paired Tactile (HPT) dataset, containing 1.6M synchronized paired frames across four vision- and array-based tactile sensors. Across distinct tactile perception and real-world manipulation tasks, HTT is shown to learn transferable representations that adapt to new tasks and previously unseen sensors. Dataset, code, and model checkpoints will be released upon publication at https://jxbi1010.github.io/htt-gh-page/.

Problem

• heterogeneous tactile sensors
• tactile representation
• cross-sensor transfer
• contact-rich manipulation
• sensor interoperability

Innovation

• Heterogeneous Tactile Transformer
• cross-modal alignment
• masked reconstruction
• tactile representation learning
• HPT dataset