🤖 AI Summary
Standard convolutions, due to their fixed structure, linearity, and reliance on local averaging, struggle to capture complex image characteristics such as low-rank structures, adaptive basis representations, and non-uniform spatial dependencies. This work proposes a unified taxonomy encompassing five classes of structured operators—decomposition-based, adaptive weighting, basis-adaptive, integral/kernel-based, and attention-based—and systematically analyzes their differences along key dimensions including locality, linearity, and equivariance. By leveraging techniques such as singular value/tensor decomposition, content-adaptive weighting, learnable analysis bases, position-dependent nonlinear kernels, and attention mechanisms, the study comprehensively evaluates the performance of these operators across image-to-image and image-to-label tasks. The findings clarify the respective strengths and limitations of each operator class, offering both theoretical insights and practical guidance for future research.
📝 Abstract
The convolution operator is the fundamental building block of modern convolutional neural networks (CNNs), owing to its simplicity, translational equivariance, and efficient implementation. However, its structure as a fixed, linear, locally-averaging operator limits its ability to capture structured signal properties such as low-rank decompositions, adaptive basis representations, and non-uniform spatial dependencies. This paper presents a systematic taxonomy of operators that extend or replace the standard convolution in learning-based image processing pipelines. We organise the landscape of alternative operators into five families: (i) decomposition-based operators, which separate structural and noise components through singular value or tensor decompositions; (ii) adaptive weighted operators, which modulate kernel contributions as a function of spatial position or signal content; (iii) basis-adaptive operators, which optimise the analysis bases together with the network weights; (iv) integral and kernel operators, which generalise the convolution to position-dependent and non-linear kernels; and (v) attention-based operators, which relax the locality assumption entirely. For each family, we provide a formal definition, a discussion of its structural properties with respect to the convolution, and a critical analysis of the tasks for which the operator is most appropriate. We further provide a comparative analysis of all families across relevant dimensions -- linearity, locality, equivariance, computational cost, and suitability for image-to-image and image-to-label tasks -- and outline the open challenges and future directions of this research area.