Difference-Driven Gating: Adaptive Feature Fusion for U-Net Decoder

2026-07-13

AI Summary

This work addresses the limited reconstruction performance in U-Net decoders caused by imprecise fusion of high- and low-level features. To overcome this, the authors propose a novel difference-driven adaptive gating mechanism that, for the first time, leverages the discrepancy between high- and low-level feature streams—rather than their content or correlation—to generate coupled gating maps that precisely modulate the fusion process. Two specific implementations are introduced: Feature Difference-based Gating (FDG), which uses the absolute difference of features for local refinement, and Entropy Difference-based Gating (EDG), which exploits signed entropy differences to enable global optimization. Extensive experiments demonstrate that the proposed approach consistently outperforms existing attention-based fusion strategies across diverse tasks, including medical image segmentation, remote sensing cloud removal, and speech separation, with EDG achieving the best overall performance.

Abstract

The U-Net style models have been widely used in many applications. A critical step in these models is to reconstruct the lower-level features using a top-down decoder. This reconstruction requires precise fusion of high-level semantics and low-level details. Existing attention-based fusion methods typically derive attention weights from the top-down decoder features (global) alone or the correlation between the top-down decoder features and the bottom-up encoder features (local), then modulate the encoder features using these weights. In this work, we explore a different paradigm: deriving attention weights from the difference between the two feature streams. To this end, we propose two difference-based gating approaches: Feature-difference gating (FDG), which directly uses the absolute difference between global and local features to generate adaptive gating maps, and Entropy-difference gating (EDG), which measures the representational certainty of each stream via information entropy and uses their signed entropy difference to derive the attention weights. Both methods produce coupled gating maps that simultaneously modulate the global and local features. Experiments on different tasks including medical image segmentation, remote sensing image cloud removal and speech separation showed that both methods outperformed existing attention-based fusion methods, and EDG performed better. The results suggested a new paradigm for multi-scale feature fusion in the U-Net style structures.

Problem

• feature fusion
• U-Net
• attention mechanism
• multi-scale representation
• decoder

Innovation

• Difference-Driven Gating
• Feature Fusion
• U-Net
• Entropy-based Attention
• Multi-scale Representation