exUMI: Extensible Robot Teaching System with Action-aware Task-agnostic Tactile Representation

2025-09-18

AI Summary

Tactile robotic learning faces critical challenges including data scarcity, sparse tactile signals, and the absence of real-time force feedback. To address these, we propose a hardware-software co-design solution. On the hardware side, we introduce the extensible Enhanced Universal Manipulation Interface (exUMI), integrating AR-based motion tracking, rotary encoders, and modular vision-tactile sensors to enable high-fidelity, fully automated calibration and scalable acquisition of over one million tactile frames—achieving 100% data usability. On the algorithmic side, we present the first action-aware Temporal Tactile Prediction (TPP) pretraining framework, which leverages self-supervised learning to acquire task-agnostic, general-purpose tactile representations. Experiments on real-world complex contact manipulation tasks demonstrate that TPP significantly outperforms conventional tactile imitation learning methods, substantially enhancing robotic dexterity and fine manipulation performance.

Technology Category

• Intelligent Robots: Manipulation
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

Tactile-aware robot learning faces critical challenges in data collection and representation due to data scarcity and sparsity, and the absence of force feedback in existing systems. To address these limitations, we introduce a tactile robot learning system with both hardware and algorithm innovations. We present exUMI, an extensible data collection device that enhances the vanilla UMI with robust proprioception (via AR MoCap and rotary encoder), modular visuo-tactile sensing, and automated calibration, achieving 100% data usability. Building on an efficient collection of over 1 M tactile frames, we propose Tactile Prediction Pretraining (TPP), a representation learning framework through action-aware temporal tactile prediction, capturing contact dynamics and mitigating tactile sparsity. Real-world experiments show that TPP outperforms traditional tactile imitation learning. Our work bridges the gap between human tactile intuition and robot learning through co-designed hardware and algorithms, offering open-source resources to advance contact-rich manipulation research. Project page: https://silicx.github.io/exUMI.

Problem

• Addresses tactile data scarcity and sparsity in robot learning
• Overcomes absence of force feedback in tactile systems
• Bridges human tactile intuition with robot manipulation learning

Innovation

• Extensible data collection device with enhanced sensing
• Tactile Prediction Pretraining framework for representation learning
• Co-designed hardware and algorithms for tactile robot learning