A Refreshed Similarity-based Upsampler for Direct High-Ratio Feature Upsampling

2024-07-02

arXiv.org

AI Summary

This paper addresses three key bottlenecks in existing similarity-driven feature upsampling methods for high-magnification (4×–16×) direct upsampling: (1) misalignment between query and key feature spaces, (2) rigid reliance on fixed inner-product similarity computation, and (3) coarse low-resolution neighborhood selection causing mosaic artifacts. To overcome these, we propose ReSFU—a robust, end-to-end trainable framework. Its core innovations include: (1) an explicit, semantically and detail-aware alignment mechanism with tunable parameters; (2) parameterized pairwise central-difference convolution for geometry-aware, dynamic similarity modeling; and (3) a fine-grained neighborhood selection strategy guided by high-resolution features. ReSFU significantly suppresses artifacts in dense prediction tasks—including semantic segmentation and depth estimation—while demonstrating strong generalization across diverse backbone architectures and straightforward deployment.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Feature Construction/Reformulation
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Feature upsampling is a fundamental and indispensable ingredient of almost all current network structures for dense prediction tasks. Recently, a popular similarity-based feature upsampling pipeline has been proposed, which utilizes a high-resolution feature as guidance to help upsample the low-resolution deep feature based on their local similarity. Albeit achieving promising performance, this pipeline has specific limitations: 1) HR query and LR key features are not well aligned; 2) the similarity between query-key features is computed based on the fixed inner product form; 3) neighbor selection is coarsely operated on LR features, resulting in mosaic artifacts. These shortcomings make the existing methods along this pipeline primarily applicable to hierarchical network architectures with iterative features as guidance and they are not readily extended to a broader range of structures, especially for a direct high-ratio upsampling. Against the issues, we meticulously optimize every methodological design. Specifically, we firstly propose an explicitly controllable query-key feature alignment from both semantic-aware and detail-aware perspectives, and then construct a parameterized paired central difference convolution block for flexibly calculating the similarity between the well-aligned query-key features. Besides, we develop a fine-grained neighbor selection strategy on HR features, which is simple yet effective for alleviating mosaic artifacts. Based on these careful designs, we systematically construct a refreshed similarity-based feature upsampling framework named ReSFU. Extensive experiments substantiate that our proposed ReSFU is finely applicable to various types of architectures in a direct high-ratio upsampling manner, and consistently achieves satisfactory performance on different dense prediction applications, showing superior generality and ease of deployment.

Problem

• Improves feature alignment in upsampling
• Enhances similarity computation flexibility
• Reduces mosaic artifacts in neighbor selection

Innovation

• Controllable feature alignment
• Parameterized similarity calculation
• Fine-grained neighbor selection