Medical Referring Image Segmentation via Next-Token Mask Prediction

2025-11-07

AI Summary

To address the challenges of architectural complexity, reliance on multimodal fusion, and multi-stage decoders in medical reference image segmentation (MRIS), this paper proposes NTP-MRISeg—a unified multimodal sequence autoregressive framework that models referential expression-guided segmentation as next-token prediction. Methodologically, it introduces three key innovations: k-next-token prediction to mitigate exposure bias, token-level contrastive learning to enhance discriminability, and memory-based hard example mining to tackle long-tailed class distributions and ambiguous lesion boundaries. The framework employs a pretrained multimodal tokenizer to jointly tokenize images, referring texts, and segmentation masks—eliminating modality-specific fusion modules and external segmentation heads. Evaluated on QaTa-COV19 and MosMedData+, NTP-MRISeg achieves state-of-the-art performance, demonstrating significant improvements in segmentation accuracy and cross-dataset generalization.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Medical Referring Image Segmentation (MRIS) involves segmenting target regions in medical images based on natural language descriptions. While achieving promising results, recent approaches usually involve complex design of multimodal fusion or multi-stage decoders. In this work, we propose NTP-MRISeg, a novel framework that reformulates MRIS as an autoregressive next-token prediction task over a unified multimodal sequence of tokenized image, text, and mask representations. This formulation streamlines model design by eliminating the need for modality-specific fusion and external segmentation models, supports a unified architecture for end-to-end training. It also enables the use of pretrained tokenizers from emerging large-scale multimodal models, enhancing generalization and adaptability. More importantly, to address challenges under this formulation-such as exposure bias, long-tail token distributions, and fine-grained lesion edges-we propose three novel strategies: (1) a Next-k Token Prediction (NkTP) scheme to reduce cumulative prediction errors, (2) Token-level Contrastive Learning (TCL) to enhance boundary sensitivity and mitigate long-tail distribution effects, and (3) a memory-based Hard Error Token (HET) optimization strategy that emphasizes difficult tokens during training. Extensive experiments on the QaTa-COV19 and MosMedData+ datasets demonstrate that NTP-MRISeg achieves new state-of-the-art performance, offering a streamlined and effective alternative to traditional MRIS pipelines.

Problem

• Streamlining multimodal fusion in medical image segmentation
• Addressing exposure bias in autoregressive mask prediction
• Enhancing boundary sensitivity for fine-grained lesion segmentation

Innovation

• Autoregressive next-token prediction for unified multimodal sequences
• Next-k token prediction to reduce cumulative errors
• Token-level contrastive learning for boundary sensitivity enhancement