VibeAct: Vibration to Actions for Contact-Rich Reactive Robot Dexterity

2026-06-25

AI Summary

Dexterous manipulation relies on high-frequency, localized contact events that are often occluded, posing significant challenges for conventional perception methods. This work proposes a tactile sensing framework based on piezoelectric microphones to capture vibroacoustic signals, enabling the construction of a shared physical representation of contact and slip dynamics usable in both real-world and simulated environments. By automatically annotating contact states from real-world data and directly training policies in simulation, the approach achieves efficient sim-to-real transfer. Integrating digital twin calibration, tactile state estimation, and simulation-based reinforcement learning, the method substantially outperforms proprioceptive and point-cloud baselines across five contact-intensive tasks—particularly excelling in those requiring continuous reactive control—and has been successfully deployed on a real dexterous hand platform, markedly improving task success rates.

Abstract

Dexterous manipulation depends on contact events that are fast, local, and often visually occluded. Piezoelectric microphones offer a compact and high-bandwidth way to sense these interactions, but the resulting vibro-acoustic signals are difficult to simulate faithfully enough for end-to-end sim-to-real policy learning on dexterous robot hands. We propose VibeAct, a framework that bridges real vibrotactile sensing and simulation-based reinforcement learning through a shared physical representation of contact and slip. In the real world, we embed piezoelectric microphones into a dexterous robot hand and collect vibro-acoustic data through teleoperation, then replay the recordings in a calibrated digital clone to automatically label per-finger contact and slip. A tactile estimator learns to predict contact and slip from real microphone waveforms, while manipulation policies are trained in simulation on the same representation computed directly from simulated contacts. This decoupling lets policies exploit rapid tactile feedback without simulating raw audio. Across five contact-rich tasks spanning regrasping, in-hand reorientation, and insertion, VibeAct consistently outperforms a proprioception-and-point-cloud baseline in simulation, with the largest gains on tasks requiring sustained reactive control, where the continuous slip-magnitude channel proves the most informative observation. The learned policies transfer to a physical dexterous hand-arm platform, improving success rates on deployed tasks. Project videos and additional details are at https://vibeact.github.io/.

Problem

• dexterous manipulation
• vibrotactile sensing
• contact-rich tasks
• sim-to-real transfer
• slip detection

Innovation

• vibrotactile sensing
• sim-to-real transfer
• dexterous manipulation
• contact and slip estimation
• reinforcement learning