RCT: A Robot-Collected Touch-Vision-Language Dataset for Tactile Generalization

2026-06-30

AI Summary

This work addresses the poor generalization of tactile perception to unseen materials by introducing a multimodal dataset comprising 29,279 frames of tactile sequences captured during full pressing interactions with 122 industrial materials across seven categories, using three DIGIT sensors. The study proposes a novel data structure preserving complete contact sequences and establishes a rigorous leave-out evaluation protocol spanning materials, categories, sensors, and sequences, revealing substantial performance degradation of existing models on novel materials. Integrating robotic autonomous data collection, tactile–visual–language alignment, contrastive learning, and a sequence-aware sampling strategy, the approach achieves a tactile-to-text Recall@1 of 25.1% ± 6.1% under the held-out material setting. Uniform sampling is shown to significantly enhance contrastive training efficacy, yielding embeddings that markedly improve category recognition for unseen materials.

Abstract

For robots manipulating open-world objects, tactile representations must generalize to unseen materials. We introduce RCT (Robotic Contact Tactile), a robot-collected touch-vision-language dataset with 29,279 tactile frames from full robot presses on 122 industrial reference materials in 7 categories, recorded with three DIGIT sensors at multiple contact positions. RCT preserves each press as a contact sequence, enabling held-out evaluation across materials, categories, sensors, contact positions, and contact sequences. Frames from one press are strongly correlated: frame-random splits can place near-duplicate observations of the same physical interaction in both training and test. With the encoder held fixed, removing contact-sequence overlap reduces tactile-to-text Recall@1 by 17.7 percentage points. When materials are additionally held out at training time, performance drops sharply, leaving held-out-material Recall@1 at 25.1 +/- 6.1% averaged over three held-out draws. The public TVL/HCT split shows the same structure: every test contact sequence appears in training, and raw-pixel nearest neighbors recover the correct sequence in 98.3% of cases. Uniformly sampling a press improves contrastive training, and RCT-trained embeddings improve category probes on unseen materials. RCT makes contact-sequence-aware, held-out-material evaluation reproducible and exposes novel-material generalization as a central challenge for robotic tactile perception. The RCT dataset is open-sourced at https://faerber-lab.github.io/RCT/

Problem

• tactile generalization
• unseen materials
• contact sequences
• robotic perception
• material generalization

Innovation

• tactile generalization
• contact sequence
• held-out material evaluation
• robot-collected dataset
• touch-vision-language