Object Tokens as a Bridge Between Segmentation and Visual Question Answering in Robotic Surgery

2026-06-14

AI Summary

This work addresses the limitations of existing surgical visual question answering (VQA) methods, which rely on coarse-grained localization (e.g., bounding boxes) and thus struggle to model fine-grained spatial relationships between instruments and tissues, hindering the reasoning capabilities of vision-language models in robotic surgery. To overcome this, the authors propose a unified framework that introduces object tokens as a bridge to jointly optimize pixel-level segmentation and VQA. Specifically, object tokens generated by a vision-language model simultaneously guide answer prediction and mask generation in a SAM decoder, enabling deep fusion of semantic and spatial information. Evaluated on RAMIE and EndoVis18, the method significantly outperforms current baselines, achieving higher VQA accuracy while providing pixel-level interpretability, thereby demonstrating the efficacy of object tokens for fine-grained understanding of surgical scenes.

Abstract

Visual Question Answering (VQA) in robotic surgery, referred to as surgical VQA, requires high-level understanding of complex surgical scenes and the integration of visual perception with language reasoning, with the potential to support surgical training and intraoperative decision-making. Recent Vision-Language Models (VLMs) have shown promising performance through parameter-efficient fine-tuning; however, most existing approaches rely on coarse visual grounding, typically limited to bounding boxes, which fails to capture the fine-grained spatial structure of surgical objects. In this work, we propose a unified framework that jointly performs pixel-level segmentation and visual question answering within a single framework. Our approach integrates a VLM with a Segment Anything Model (SAM)-based decoder and represents scene elements as object tokens generated by the VLM. These object tokens guide answer prediction and are further projected to the SAM-based decoder to produce segmentation masks. By optimizing the object token embeddings through both segmentation and question answering objectives, the model learns spatially grounded representations that enhance visual reasoning while providing explicit pixel-level grounding. We evaluate the proposed method on the private RAMIE (Robot-Assisted Minimally Invasive Esophagectomy) dataset and the public EndoVis18 dataset, where it consistently outperforms baseline methods for surgical VQA. These results demonstrate that incorporating context-aware object tokens into vision-language models improves fine-grained surgical scene understanding.

Problem

• Surgical Visual Question Answering
• Fine-grained Spatial Understanding
• Visual Grounding
• Robotic Surgery
• Vision-Language Models

Innovation

• Object Tokens
• Surgical Visual Question Answering
• Pixel-level Segmentation
• Vision-Language Models
• Spatial Grounding