Anisotropic Pooling for LUT-realizable CNN Image Restoration

2025-10-24

AI Summary

To address the limitation of isotropic average pooling in LUT-based CNNs for image restoration—its inability to adapt to the anisotropic structures inherent in natural images—this paper proposes an anisotropic pooling strategy. Methodologically, it (1) replaces average pooling with generalized median pooling to better preserve edges and textures; (2) introduces a data-driven, adaptive directional weighting mechanism that dynamically fuses lookup table (LUT) outputs across multiple orientations; and (3) constructs an end-to-end trainable patch-wise LUT-CNN architecture. Evaluated on multiple image restoration benchmarks—including denoising, super-resolution, and JPEG artifact removal—the proposed method consistently outperforms existing LUT-based approaches, achieving superior performance in PSNR, LPIPS, and perceptual quality. These results empirically validate that anisotropic modeling is critical to enhancing the representational capacity and practical efficacy of LUT-parameterized neural networks.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Learning to Search
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Table look-up realization of image restoration CNNs has the potential of achieving competitive image quality while being much faster and resource frugal than the straightforward CNN implementation. The main technical challenge facing the LUT-based CNN algorithm designers is to manage the table size without overly restricting the receptive field. The prevailing strategy is to reuse the table for small pixel patches of different orientations (apparently assuming a degree of isotropy) and then fuse the look-up results. The fusion is currently done by average pooling, which we find being ill suited to anisotropic signal structures. To alleviate the problem, we investigate and discuss anisotropic pooling methods to replace naive averaging for improving the performance of the current LUT-realizable CNN restoration methods. First, we introduce the method of generalized median pooling which leads to measurable gains over average pooling. We then extend this idea by learning data-dependent pooling coefficients for each orientation, so that they can adaptively weigh the contributions of differently oriented pixel patches. Experimental results on various restoration benchmarks show that our anisotropic pooling strategy yields both perceptually and numerically superior results compared to existing LUT-realizable CNN methods.

Problem

• Managing table size while preserving receptive field in LUT-based CNNs
• Replacing average pooling with anisotropic methods for better fusion
• Improving restoration performance using data-adaptive pooling coefficients

Innovation

• Anisotropic pooling replaces average pooling
• Generalized median pooling improves restoration performance
• Learned coefficients adaptively weigh oriented patches