SegDINO: An Efficient Design for Medical and Natural Image Segmentation with DINO-V3

2025-08-31

AI Summary

To address the high parameter count and computational overhead of DINO-based self-supervised models in image segmentation—caused by heavy decoders, explicit multi-scale fusion, or complex learnable upsampling—this paper proposes an efficient lightweight segmentation framework. It freezes the DINOv3 backbone, introduces a feature resolution and channel alignment module to fuse multi-level features, and replaces conventional convolutional upsampling with a pure MLP-based lightweight decoder. By eliminating explicit multi-scale fusion and learnable upsampling operations, the method significantly reduces trainable parameters (averaging 72% fewer than state-of-the-art methods) while preserving strong representational capacity. Extensive experiments demonstrate state-of-the-art performance on three medical and three natural image segmentation benchmarks, validating both generalizability and practicality. The code is publicly available.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Unsupervised & Self-Supervised Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

The DINO family of self-supervised vision models has shown remarkable transferability, yet effectively adapting their representations for segmentation remains challenging. Existing approaches often rely on heavy decoders with multi-scale fusion or complex upsampling, which introduce substantial parameter overhead and computational cost. In this work, we propose SegDINO, an efficient segmentation framework that couples a frozen DINOv3 backbone with a lightweight decoder. SegDINO extracts multi-level features from the pretrained encoder, aligns them to a common resolution and channel width, and utilizes a lightweight MLP head to directly predict segmentation masks. This design minimizes trainable parameters while preserving the representational power of foundation features. Extensive experiments across six benchmarks, including three medical datasets (TN3K, Kvasir-SEG, ISIC) and three natural image datasets (MSD, VMD-D, ViSha), demonstrate that SegDINO consistently achieves state-of-the-art performance compared to existing methods. Code is available at https://github.com/script-Yang/SegDINO.

Problem

• Efficiently adapts DINOv3 for segmentation tasks
• Reduces parameter overhead and computational costs
• Achieves state-of-the-art performance across medical and natural images

Innovation

• Frozen DINOv3 backbone with lightweight decoder
• Aligns multi-level features to common resolution
• Lightweight MLP head predicts segmentation masks directly