HT-Bench: Benchmarking and Learning Dexterous Full-Hand Tactile Representations with Egocentric Vision

2026-06-17

AI Summary

This work addresses the lack of a unified benchmark in tactile representation learning, which has been hindered by the diversity of sensor designs, data formats, and robotic morphologies. The authors propose HT-Bench, the first large-scale, whole-hand visuo-tactile multi-task benchmark tailored for dexterous manipulation, along with HandTouch, a novel encoder that jointly learns RGB images and tactile frames through a progressive spatial–cross-modal–temporal training strategy. Evaluated across four tasks, the method substantially outperforms existing baselines: achieving 85.23% Recall@5 in fine-grained retrieval, reducing RMSE to 0.010 in tactile inpainting, and attaining a cIoU of 0.705 in out-of-domain visual-to-tactile synthesis. These results demonstrate significant improvements in contact geometry modeling, cross-modal alignment, and task generalization capabilities of learned tactile representations.

Abstract

Establishing a universal benchmark for tactile representation learning in robotic manipulation remains challenging due to the diversity of tactile sensor designs, data formats, and robot embodiments. Rather than seeking to establish such, we explore a scalable and promising direction for future development: egocentric vision paired with full-hand tactile data. To this end, we introduce \textbf{HT-Bench}, a large-scale multi-task benchmark for dexterous full-hand tactile sensing, comprising 10M RGB frames and 7.8M tactile frames collected across 226 tasks. HT-Bench evaluates tactile representations from three key perspectives: whether they encode meaningful contact geometry, whether they can align tactile observations with visual information, and whether they generalize to unseen tasks. To assess these capabilities, HT-Bench includes four tasks: fine-grained tactile similarity retrieval, masked tactile inpainting, vision-to-tactile synthesis, and multimodal tactile frame prediction. We further propose \textbf{HandTouch}, a vector-quantized vision--tactile encoder that learns tactile representations through progressive spatial, cross-modal, and temporal training. Across HT-Bench, HandTouch consistently outperforms representative tactile encoder baselines, improving Recall@5 on fine-grained tactile similarity retrieval from 74.65\% to 85.23\%, reducing RMSE on masked tactile inpainting from 0.022 to 0.010, and increasing OOD cIoU on vision-to-tactile synthesis from 0.628 to 0.705. These results demonstrate the effectiveness of HandTouch and suggest that large-scale egocentric full-hand tactile data provides a scalable basis for evaluating and advancing tactile representation learning in dexterous manipulation.

Problem

• tactile representation learning
• robotic manipulation
• benchmarking
• egocentric vision
• full-hand tactile sensing

Innovation

• tactile representation learning
• egocentric vision
• full-hand tactile sensing
• multimodal perception
• dexterous manipulation