Phase-and-First-Arrival VLM Feedback for Sparse-Reward Reinforcement Learning in Surgical Manipulation

2026-09-07

AI Summary

研究解决了稀疏奖励强化学习中手术操作反馈不足的问题，通过引入阶段和首次到达反馈方法，利用视觉-语言模型评估任务进展，提高学习效率。

Abstract

Sparse outcome feedback limits what robots can learn from unsuccessful attempts at complex manipulation. Failed multi-stage surgical attempts can contain grasps, lifts, or transfers worth reusing. In sparse-reward reinforcement learning, terminal rewards collapse such attempts to the same outcome, while scalar vision-language model (VLM) ratings reveal neither what progress merits credit nor when it occurred. We introduce phase-and-first-arrival feedback: one VLM query per recorded episode identifies the furthest visually verified task phase and when that phase is first reached, allowing the learner to reuse partial behavior and localize credit. We instantiate it in SurgPhaseBench, a phase-structured suite spanning rigid and deformable tasks, and evaluate it in simulation and hardware. Across five simulated tasks, our method reaches 75.2% mean success, compared with 52.1% for a reward based on Contrastive Language-Image Pre-training (CLIP) using the same visual input; the advantage persists when only the feedback representation changes. On hardware, the same record supports autonomous block picking and slip recovery. Together, these results show that trajectory-level visual supervision can preserve partial progress while providing the temporal credit needed for sparse-reward control.

Problem

• sparse-reward reinforcement learning
• surgical manipulation
• visual-language model
• partial progress
• temporal credit

Innovation

• Phase-and-First-Arrival Feedback
• Sparse-Reward Reinforcement Learning
• Surgical Manipulation
• VLM Query
• Temporal Credit