In-the-Wild Compliant Manipulation with UMI-FT

2026-01-15

AI Summary

Commercial force/torque sensors are often expensive, bulky, and fragile, hindering their use in large-scale learning of force-aware manipulation strategies. To address this, this work proposes UMI-FT, a lightweight, low-cost, wearable fingertip force-sensing platform that integrates miniature six-axis force/torque sensors on each finger to simultaneously capture RGB-D data, pose, and per-finger force information. By combining multimodal fusion with demonstration-based reinforcement learning, the system trains an end-to-end adaptive compliance policy capable of real-time prediction of position targets, grasp forces, and stiffness. Evaluated on tasks including whiteboard wiping, zucchini piercing, and lightbulb insertion, the proposed policy significantly outperforms baselines lacking compliance or force perception, achieving fine-grained control over both external contact forces and internal grasp forces.

Technology Category

• Intelligent Robots: Manipulation
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Many manipulation tasks require careful force modulation. With insufficient force the task may fail, while excessive force could cause damage. The high cost, bulky size and fragility of commercial force/torque (F/T) sensors have limited large-scale, force-aware policy learning. We introduce UMI-FT, a handheld data-collection platform that mounts compact, six-axis force/torque sensors on each finger, enabling finger-level wrench measurements alongside RGB, depth, and pose. Using the multimodal data collected from this device, we train an adaptive compliance policy that predicts position targets, grasp force, and stiffness for execution on standard compliance controllers. In evaluations on three contact-rich, force-sensitive tasks (whiteboard wiping, skewering zucchini, and lightbulb insertion), UMI-FT enables policies that reliably regulate external contact forces and internal grasp forces, outperforming baselines that lack compliance or force sensing. UMI-FT offers a scalable path to learning compliant manipulation from in-the-wild demonstrations. We open-source the hardware and software to facilitate broader adoption at:https://umi-ft.github.io/.

Problem

• compliant manipulation
• force modulation
• in-the-wild demonstration
• force/torque sensing
• dexterous manipulation

Innovation

• compliant manipulation
• force/torque sensing
• in-the-wild demonstration
• adaptive compliance policy
• multimodal data collection