Attribute Retrieving for Open-Vocabulary Endoscopic Compositional Referring Segmentation

2026-07-09

AI Summary

This work addresses the challenge of open-vocabulary referring expression segmentation in endoscopic images, which is hindered by scarce annotations and complex text-image relationships. To enhance the model’s comprehension of open-vocabulary instructions, the authors propose AR-ERIS, a framework augmented with fine-grained attribute retrieval. They also introduce ReferEndoscopy, the first large-scale benchmark for endoscopic referring expression segmentation, and leverage vision-language pre-trained models by pre-training and fine-tuning on this dataset. Experimental results demonstrate that the proposed method achieves state-of-the-art performance on both synthetic and real endoscopic data, significantly improving segmentation accuracy and generalization capability.

Abstract

Referring Image Segmentation (RIS) aims to segment image regions specified by natural language, enabling fine-grained and controllable visual understanding. Extending RIS to endoscopic imagery, however, presents unique challenges, including scarce high-quality annotations and complex, domain-specific image-text relationships. Although recent vision-language models demonstrate strong cross-domain alignment, they often fail to capture fine-grained textual cues in endoscopic settings, resulting in suboptimal performance and limited generalization. To address these challenges, we introduce ReferEndoscopy, a large-scale benchmark for RIS in the endoscopy field. Building on this dataset, we propose the Attribute Retrieval-based Endoscopic-RIS (AR-ERIS) framework for open-vocabulary endoscopic compositional referring segmentation. AR-ERIS leverages attribute retrieval for open-vocabulary endoscopic compositional referring segmentation and is pretrained on the curated ReferEndoscopy dataset, achieving state-of-the-art performance with strong generalization across both simulated and real-world endoscopic data. The dataset and code will be publicly released upon completion of the review process.

Problem

• Referring Image Segmentation
• Endoscopic Imaging
• Open-Vocabulary
• Compositional Referring
• Attribute Retrieval

Innovation

• Attribute Retrieval
• Open-Vocabulary Segmentation
• Endoscopic Referring Segmentation
• Vision-Language Model
• Compositional Referring