VTouch++: A Multimodal Dataset with Vision-Based Tactile Enhancement for Bimanual Manipulation

2026-04-22

AI Summary

This work addresses the limitations of dual-arm robotic manipulation in contact-rich tasks, which stem from the lack of high-quality interaction signals, loosely structured task formulations, and insufficient data scale. To overcome these challenges, the authors introduce a large-scale multimodal dataset that uniquely integrates high-fidelity visuo-tactile perception, a matrix-based task design, and an automated data collection pipeline. The dataset enables generalization across diverse robotic platforms, control strategies, and manipulation tasks. Its utility and effectiveness are validated through cross-modal retrieval experiments and real-world robot deployments, demonstrating strong transferability and robustness in practical scenarios.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Embodied intelligence has advanced rapidly in recent years; however, bimanual manipulation-especially in contact-rich tasks remains challenging. This is largely due to the lack of datasets with rich physical interaction signals, systematic task organization, and sufficient scale. To address these limitations, we introduce the VTOUCH dataset. It leverages vision based tactile sensing to provide high-fidelity physical interaction signals, adopts a matrix-style task design to enable systematic learning, and employs automated data collection pipelines covering real-world, demand-driven scenarios to ensure scalability. To further validate the effectiveness of the dataset, we conduct extensive quantitative experiments on cross-modal retrieval as well as real-robot evaluation. Finally, we demonstrate real-world performance through generalizable inference across multiple robots, policies, and tasks.

Problem

• bimanual manipulation
• contact-rich tasks
• multimodal dataset
• physical interaction signals
• embodied intelligence

Innovation

• vision-based tactile sensing
• bimanual manipulation
• multimodal dataset
• matrix-style task design
• automated data collection