SegDINO: Introducing Multi-Scale Structure into DINO for Efficient Medical Image Segmentation

2026-06-16

AI Summary

This work addresses the limited efficacy of directly applying the self-supervised DINO model to medical image segmentation and the high computational and parameter overhead incurred by mainstream approaches that rely on heavy decoders. To overcome these limitations, the authors propose SegDINO, a novel framework that integrates a DINOv3 backbone with a lightweight multi-scale modeling strategy, demonstrating that multi-scale architecture design is more critical than decoder capacity. By introducing Token Pyramid Adaptation (TPA) and Scale-Aware Decoding (SAD), SegDINO efficiently reconstructs and fuses multi-scale features, significantly enhancing small lesion segmentation while maintaining model compactness. Extensive experiments on the newly curated PanCT pancreatic tumor CT dataset and three public medical segmentation benchmarks show that SegDINO achieves state-of-the-art performance, offering an optimal balance between accuracy and computational efficiency.

Abstract

Self-supervised DINO models provide strong transferable visual representations, yet applying them directly to image segmentation remains challenging. Existing approaches commonly rely on heavy decoders with complex upsampling, introducing substantial parameter and computational overhead. We observe that introducing scale into DINO features is far more critical than increasing decoder capacity. In this work, we present SegDINO, an efficient segmentation framework that integrates a DINOv3 backbone with lightweight scale modeling. SegDINO introduces Token Pyramid Adaptation (TPA) to reorganize intermediate DINO features into a pseudo multi-scale hierarchy, and Scale-Aware Decoding (SAD) for efficient intra-scale refinement and top-down multi-scale propagation. We further curate PanCT, a new CT dataset containing 284 patients with expert-annotated pancreatic tumors, to assess SegDINO's ability to handle difficult small-lesion cases. Extensive experiments on PanCT and three public benchmarks demonstrate that SegDINO achieves state-of-the-art results with high efficiency. The code is available at https://github.com/script-Yang/segdino_v2.

Problem

• medical image segmentation
• self-supervised learning
• DINO
• small lesion segmentation
• efficient segmentation

Innovation

• multi-scale modeling
• self-supervised learning
• medical image segmentation
• lightweight decoder
• DINO