BiliVLA: Scene-Aware Vision-Language-Action Model with Reinforcement Learning for Autonomous Biliary Endoscopic Navigation

2026-06-22

AI Summary

This study addresses the challenges of autonomous navigation and precise cannulation during ERCP procedures, which are hindered by anatomical variations, specular reflections, occlusions, and tissue contact. To overcome these issues, the authors propose a vision–language–action learning framework conditioned on high-level instructions. The approach integrates scene-aware supervision and a safety recovery mechanism, combining semantic target grounding with three-degree-of-freedom discrete motion control to align perception and action through a two-stage training paradigm. Innovatively fusing semantic bounding-box grounding, supervised fine-tuning (SFT), and grouped relative policy optimization (GRPO), the method significantly enhances decision consistency and action reliability. In ex vivo phantom experiments, the system achieved an average action accuracy of 91.96% across three ERCP subtasks and an overall success rate of 84.85%, demonstrating robust autonomous navigation capabilities in complex clinical scenarios.

Abstract

Endoscopic retrograde cholangiopancreatography (ERCP) demands precise endoscopic navigation and stable biliary cannulation within a narrow monocular field characterized by specular reflections, partial occlusions, and frequent tissue contact. Although recent robotic systems and vision-based assistance techniques improve operator ergonomics and provide perceptual cues, their performance degrades under pronounced anatomical variability and safety-critical visual artifacts, which hinders reliable autonomy in cannulation-grade procedures. Here, we present BiliVLA, a scene-aware Vision-Language-Action (VLA) framework that formulates biliary endoscopic navigation as an instruction-conditioned visuomotor learning problem. Given an endoscopic observation and a stage-specific language instruction, BiliVLA jointly predicts the target category, a grounded bounding box, and a discrete three degrees of freedom (DoF) motor command for a continuum endoscope. The proposed framework incorporates scene-aware supervision to enhance semantic target consistency and safety-aware recovery supervision to induce conservative retreat behaviors under luminal wall contact. A key component of BiliVLA is a two-stage training paradigm that combines grounding-enhanced supervised fine-tuning (SFT) with Group Relative Policy Optimization (GRPO), which significantly improves action reliability and decision consistency during closed-loop navigation. Across three ERCP subtasks, BiliVLA achieves an average action precision of 91.96\% and an overall success rate (SR) of 84.85\% in real-world phantom experiments. These results indicate that integrating semantic grounding, scene-aware learning, and reward-guided optimization improves perception-action alignment and enables robust autonomous endoscopic navigation.

Problem

• ERCP
• biliary cannulation
• autonomous endoscopic navigation
• visual artifacts
• anatomical variability

Innovation

• Vision-Language-Action (VLA)
• scene-aware supervision
• semantic grounding
• reinforcement learning
• endoscopic navigation