Hybrid Offline-Online Reinforcement Learning for Sensorless, High-Precision Force Regulation in Surgical Robotic Grasping

2026-02-27

AI Summary

This work addresses the challenge of high-precision, sensorless grasp-force control in tendon-driven surgical instruments, where complex nonlinear couplings arise from motor dynamics, transmission compliance, friction, and end-effector mechanics. To overcome this, the authors propose a novel paradigm integrating physics-consistent modeling with a hybrid offline–online reinforcement learning framework. A high-fidelity differential-algebraic digital twin of the da Vinci Xi grasper is developed, and a three-stage learning pipeline is employed: receding-horizon CMA-ES generates expert trajectories, Implicit Q-Learning (IQL) initializes the policy offline, and TD3 fine-tunes it online. Relying solely on proximal measurements—without distal force sensors—the method achieves sub-1% tracking error for multi-harmonic force profiles in simulation and under 4% average error in hardware experiments. The resulting 71k-parameter policy enables real-time deployment at kHz rates and demonstrates excellent sim-to-real transferability.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Precise grasp force regulation in tendon-driven surgical instruments is fundamentally limited by nonlinear coupling between motor dynamics, transmission compliance, friction, and distal mechanics. Existing solutions typically rely on distal force sensing or analytical compensation, increasing hardware complexity or degrading performance under dynamic motion. We present a sensorless control framework that combines physics-consistent modeling and hybrid reinforcement learning to achieve high-precision distal force regulation in a proximally actuated surgical end-effector. We develop a first-principles digital twin of the da Vinci Xi grasping mechanism that captures coupled electrical, transmission, and jaw dynamics within a unified differential-algebraic formulation. To safely learn control policies in this stiff and highly nonlinear system, we introduce a three-stage pipeline:(i)a receding-horizon CMA-ES oracle that generates dynamically feasible expert trajectories,(ii)fully offline policy learning via Implicit Q-Learning to ensure stable initialization without unsafe exploration, and (iii)online refinement using TD3 for adaptation to on-policy dynamics. The resulting policy directly maps proximal measurements to motor voltages and requires no distal sensing. In simulation, the controller maintains grasp force within 1% of the desired reference during multi-harmonic jaw motion. Hardware experiments demonstrate average force errors below 4% across diverse trajectories, validating sim-to-real transfer. The learned policy contains approximately 71k param and executes at kH rates, enabling real-time deployment. These results demonstrate that high-fidelity modeling combined with structured offline-online RL can recover precise distal force behavior without additional sensing, offering a scalable and mechanically compatible solution for surgical robotic manipulation.

Problem

• force regulation
• sensorless control
• surgical robotics
• tendon-driven systems
• nonlinear coupling

Innovation

• sensorless force control
• hybrid offline-online reinforcement learning
• digital twin
• surgical robotics
• Implicit Q-Learning