GazeRefine: Expert Gaze as a Test-Time Prompt for Training-Free Medical Image Segmentation

2026-09-01

AI Summary

GazeRefine利用专家注视点作为测试时的提示，实现无需训练的医学图像分割，解决了依赖密集标注和任务特定训练的问题。

Abstract

Medical image segmentation remains difficult to scale because high-performing methods typically rely on dense expert annotations and task-specific training. We introduce GazeRefine, a training-free framework that uses gaze as an inference-time prompt for zero-shot medical image segmentation. Sparse, duration-weighted fixations are converted into foreground and background priors that initialize semantic prototypes in frozen DINOv3 feature space. These prototypes are iteratively refined through foreground-background discrimination, feature-space affinity propagation, and anchoring to the initial gaze guidance, allowing segmentation to extend beyond directly fixated regions while limiting semantic drift. GazeRefine requires no segmentation masks, fine-tuning, adapters, prompt encoders, or gradient updates. We evaluate the method on gaze-annotated polyp segmentation and prostate MRI segmentation. The results show strong performance on colonoscopy images and competitive performance on prostate MRI, supporting gaze-guided prototype refinement as a promising approach for segmentation-label-efficient, human-in-the-loop medical image segmentation. Our tools and code can be found in the following repository: https://github.com/MohammedOussamaBEN/GazeRefine.git

Problem

• medical image segmentation
• expert annotations
• task-specific training

Innovation

• gaze as prompt
• training-free framework
• zero-shot medical image segmentation
• semantic prototype refinement
• DINOv3 feature space