UniT: Data Efficient Tactile Representation with Generalization to Unseen Objects

2024-08-12

AI Summary

This work addresses the challenge of few-shot cross-task zero-shot transfer in tactile learning. We propose UniT—the first method to learn general-purpose tactile representations from a single, simple-object tactile image. UniT leverages a VQGAN-based architecture to construct a compact, disentangled latent representation space for tactile data, requiring no task-specific annotations or additional fine-tuning. The learned representations enable direct zero-shot transfer to diverse downstream tasks, including tactile perception (e.g., in-hand 3D/6D pose estimation and tactile classification) and manipulation policy learning. Compared to existing visual and tactile representation learning approaches, UniT achieves state-of-the-art performance across multiple benchmarks. It is successfully deployed on three real-world robotic manipulation tasks, demonstrating key advantages: plug-and-play usability, extreme data efficiency (single-object training), and task-agnostic generalization.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Representation Learning for Vision
• Machine Learning: Multimodal Learning

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

UniT is an approach to tactile representation learning, using VQGAN to learn a compact latent space and serve as the tactile representation. It uses tactile images obtained from a single simple object to train the representation with generalizability. This tactile representation can be zero-shot transferred to various downstream tasks, including perception tasks and manipulation policy learning. Our benchmarkings on in-hand 3D pose and 6D pose estimation tasks and a tactile classification task show that UniT outperforms existing visual and tactile representation learning methods. Additionally, UniT's effectiveness in policy learning is demonstrated across three real-world tasks involving diverse manipulated objects and complex robot-object-environment interactions. Through extensive experimentation, UniT is shown to be a simple-to-train, plug-and-play, yet widely effective method for tactile representation learning. For more details, please refer to our open-source repository https://github.com/ZhengtongXu/UniT and the project website https://zhengtongxu.github.io/unit-website/.

Problem

• Learning compact tactile representations from minimal object data
• Zero-shot transfer to diverse perception and manipulation tasks
• Outperforming existing methods in pose estimation and classification

Innovation

• Uses VQGAN for compact tactile representation
• Trains with single object for generalizability
• Zero-shot transfer to diverse downstream tasks