Trajectory Optimization for In-Hand Manipulation with Tactile Force Control

2025-03-11

AI Summary

Dexterous hands exhibit limited dexterity and poor robustness in in-hand manipulation of small objects. Method: This paper proposes a closed-loop trajectory optimization framework integrating magnetic tactile sensors (MTS) with nonlinear programming (NLP). It innovatively embeds tactile force feedback control and contact-state-aware object pose estimation directly into the NLP formulation, jointly modeling compliance and dynamic contact-point migration to enable high-precision finger motion planning and real-time force–position coordinated control during rolling manipulation. Results: Evaluated on the Shadow Dexterous Hand, the method achieves a 30% improvement in rolling manipulation success rate over open-loop baselines, significantly enhancing localization accuracy and environmental adaptability. It provides a scalable, perception–planning–control integrated solution for fine manipulation of small objects.

Technology Category

• Intelligent Robots: Manipulation
• Search and Optimization: Learning to Search
• Computer Vision: Motion & Tracking

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

The strength of the human hand lies in its ability to manipulate small objects precisely and robustly. In contrast, simple robotic grippers have low dexterity and fail to handle small objects effectively. This is why many automation tasks remain unsolved by robots. This paper presents an optimization-based framework for in-hand manipulation with a robotic hand equipped with compact Magnetic Tactile Sensors (MTSs). The small form factor of the robotic hand from Shadow Robot introduces challenges in estimating the state of the object while satisfying contact constraints. To address this, we formulate a trajectory optimization problem using Nonlinear Programming (NLP) for finger movements while ensuring contact points to change along the geometry of the fingers. Using the optimized trajectory from the solver, we implement and test an open-loop controller for rolling motion. To further enhance robustness and accuracy, we introduce a force controller for the fingers and a state estimator for the object utilizing MTSs. The proposed framework is validated through comparative experiments, showing that incorporating the force control with compliance consideration improves the accuracy and robustness of the rolling motion. Rolling an object with the force controller is 30% more likely to succeed than running an open-loop controller. The demonstration video is available at https://youtu.be/6J_muL_AyE8.

Problem

• Optimizing robotic hand trajectory for precise in-hand manipulation.
• Enhancing object state estimation with compact Magnetic Tactile Sensors.
• Improving rolling motion accuracy using force control and compliance.

Innovation

• Optimization-based framework for in-hand manipulation
• Magnetic Tactile Sensors for state estimation
• Force controller enhances rolling motion accuracy