UniTac: A Unified Multimodal Model for Cross-Sensor Tactile Understanding and Generation

2026-06-30

AI Summary

Existing unified multimodal models struggle to handle the complexity of tactile signals, whose perceptual meaning is jointly determined by object semantics and sensor configuration, and lack the capacity for cross-sensor understanding and generation. This work proposes the first tactile-oriented unified multimodal framework, modeling touch as a transition from non-contact to contact states. It introduces a dual-level representation capturing both object properties and sensor configurations, combined with a two-stage training strategy—reconstruction followed by alignment—and a sensor-prior-informed sampling approach to unify tactile understanding and generation. Trained on large-scale, multi-sensor tactile data, the method achieves state-of-the-art performance on tactile understanding tasks and generates highly realistic cross-sensor tactile signals.

Abstract

Unified multimodal models (UMMs) have shown great promise in integrating understanding and generation across diverse modalities. However, existing research rarely extends this paradigm to the tactile domain, where both object-level semantics and sensor-level configurations jointly determine the meaning of touch. To address this gap, we propose UniTac, the first UMM designed for tactile understanding and generation. UniTac models the tactile process as a transition from non-contact to contact, capturing the physical interaction between sensors and objects through a dual-level representation that encodes both sensor and object attributes. For tactile understanding, UniTac introduces two tasks, object property description and sensor identification, to enhance reasoning over physical and cross-sensor information. For tactile generation, we design a two-stage training paradigm consisting of reconstruction and alignment, together with a sensor-prior-based sampling strategy that simulates realistic tactile contact. Trained on large-scale multi-sensor datasets, UniTac achieves state-of-the-art performance in tactile understanding and generates realistic tactile signals across sensors.

Problem

• tactile understanding
• tactile generation
• multimodal model
• cross-sensor
• unified model

Innovation

• unified multimodal model
• tactile understanding
• tactile generation
• dual-level representation
• sensor-prior sampling