Enhancing Feature Fusion of U-like Networks with Dynamic Skip Connections

2025-09-18

AI Summary

U-Net’s static skip connections suffer from two key limitations: (i) inter-feature constraints—lacking content-aware cross-layer feature interaction—and (ii) intra-feature constraints—insufficient multi-scale feature aggregation. To address these, we propose the Dynamic Skip Connection Module (DSCM), which jointly incorporates test-time training (TTT) and dynamic multi-scale kernels (DMSK) to enable content-adaptive fusion of high- and low-level features and global-context-guided multi-scale interaction. DSCM is architecture-agnostic and seamlessly integrates into diverse U-shaped backbones—including CNNs, Transformers, and Mamba-based models. Evaluated across multiple medical image segmentation benchmarks, DSCM consistently improves segmentation accuracy while demonstrating strong generalizability and robustness to domain shifts and annotation noise.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Calibration & Uncertainty Quantification
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

U-like networks have become fundamental frameworks in medical image segmentation through skip connections that bridge high-level semantics and low-level spatial details. Despite their success, conventional skip connections exhibit two key limitations: inter-feature constraints and intra-feature constraints. The inter-feature constraint refers to the static nature of feature fusion in traditional skip connections, where information is transmitted along fixed pathways regardless of feature content. The intra-feature constraint arises from the insufficient modeling of multi-scale feature interactions, thereby hindering the effective aggregation of global contextual information. To overcome these limitations, we propose a novel Dynamic Skip Connection (DSC) block that fundamentally enhances cross-layer connectivity through adaptive mechanisms. The DSC block integrates two complementary components. (1) Test-Time Training (TTT) module. This module addresses the inter-feature constraint by enabling dynamic adaptation of hidden representations during inference, facilitating content-aware feature refinement. (2) Dynamic Multi-Scale Kernel (DMSK) module. To mitigate the intra-feature constraint, this module adaptively selects kernel sizes based on global contextual cues, enhancing the network capacity for multi-scale feature integration. The DSC block is architecture-agnostic and can be seamlessly incorporated into existing U-like network structures. Extensive experiments demonstrate the plug-and-play effectiveness of the proposed DSC block across CNN-based, Transformer-based, hybrid CNN-Transformer, and Mamba-based U-like networks.

Problem

• Enhancing feature fusion in U-like medical image networks
• Overcoming static inter-feature constraints in skip connections
• Addressing insufficient multi-scale feature interaction modeling

Innovation

• Dynamic Skip Connection adaptive cross-layer connectivity
• Test-Time Training for content-aware feature refinement
• Dynamic Multi-Scale Kernel for context integration