TLA: Tactile-Language-Action Model for Contact-Rich Manipulation

2025-03-11

AI Summary

To address the lack of joint tactile-language-action modeling in haptically intensive, language-guided robotic manipulation tasks—such as fingertip peg-in-hole assembly—this paper proposes the first language-tactile-action trimodal alignment framework, leveraging language anchoring for semantic interpretation of tactile sequences and robust policy generation. Methodologically, we design a tactile sequence encoder and a language-guided cross-modal attention mechanism, integrated into an end-to-end action decoder. We also introduce the first large-scale, 24K-sample dataset pairing fingertip tactile signals with natural-language instructions. Evaluation shows >85% success rate on unseen peg geometries and insertion gaps—significantly outperforming diffusion-based policies and other baselines—while achieving substantial gains in both action accuracy and task effectiveness.

Technology Category

• Intelligent Robots: Manipulation
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Language and Vision

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Significant progress has been made in vision-language models. However, language-conditioned robotic manipulation for contact-rich tasks remains underexplored, particularly in terms of tactile sensing. To address this gap, we introduce the Tactile-Language-Action (TLA) model, which effectively processes sequential tactile feedback via cross-modal language grounding to enable robust policy generation in contact-intensive scenarios. In addition, we construct a comprehensive dataset that contains 24k pairs of tactile action instruction data, customized for fingertip peg-in-hole assembly, providing essential resources for TLA training and evaluation. Our results show that TLA significantly outperforms traditional imitation learning methods (e.g., diffusion policy) in terms of effective action generation and action accuracy, while demonstrating strong generalization capabilities by achieving over 85% success rate on previously unseen assembly clearances and peg shapes. We publicly release all data and code in the hope of advancing research in language-conditioned tactile manipulation skill learning. Project website: https://sites.google.com/view/tactile-language-action/

Problem

• Develops a model for tactile-language-action integration in robotics.
• Addresses lack of tactile sensing in language-conditioned robotic manipulation.
• Improves action generation and accuracy in contact-rich tasks.

Innovation

• TLA model integrates tactile feedback with language grounding.
• Dataset includes 24k tactile action instruction pairs.
• TLA outperforms traditional methods in action accuracy.