LFA applied to CNNs: Efficient Singular Value Decomposition of Convolutional Mappings by Local Fourier Analysis

2025-06-05

AI Summary

Computing the full singular value spectrum of large-scale convolutional operators is computationally prohibitive; conventional FFT-based methods are limited by O(N log N) complexity and cannot guarantee exact spectral recovery. Method: We propose a linear-time algorithm grounded in Local Fourier Analysis (LFA), uniquely integrating LFA with the translation-invariance property of convolutional operators to construct an exact frequency-domain analytical model. Contribution/Results: The method computes all singular values and corresponding singular vectors in O(N) time, with rigorous theoretical guarantees and natural extensibility to high-dimensional and multi-channel convolutions. Experiments demonstrate that it overcomes hardware memory and computational bottlenecks, enabling exact full-spectrum decomposition for arbitrary input sizes and channel counts. It achieves efficient, practical deployment in downstream tasks including model compression and robustness analysis.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Computer Vision: Large Vision Models
• Search and Optimization: Non-convex Optimization

Application Category

• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

The singular values of convolutional mappings encode interesting spectral properties, which can be used, e.g., to improve generalization and robustness of convolutional neural networks as well as to facilitate model compression. However, the computation of singular values is typically very resource-intensive. The naive approach involves unrolling the convolutional mapping along the input and channel dimensions into a large and sparse two-dimensional matrix, making the exact calculation of all singular values infeasible due to hardware limitations. In particular, this is true for matrices that represent convolutional mappings with large inputs and a high number of channels. Existing efficient methods leverage the Fast Fourier transformation (FFT) to transform convolutional mappings into the frequency domain, enabling the computation of singular values for matrices representing convolutions with larger input and channel dimensions. For a constant number of channels in a given convolution, an FFT can compute N singular values in O(N log N) complexity. In this work, we propose an approach of complexity O(N) based on local Fourier analysis, which additionally exploits the shift invariance of convolutional operators. We provide a theoretical analysis of our algorithm's runtime and validate its efficiency through numerical experiments. Our results demonstrate that our proposed method is scalable and offers a practical solution to calculate the entire set of singular values - along with the corresponding singular vectors if needed - for high-dimensional convolutional mappings.

Problem

• Efficiently compute singular values for high-dimensional CNNs
• Overcome resource-intensive SVD in convolutional mappings
• Exploit shift invariance to reduce computational complexity

Innovation

• Local Fourier Analysis for efficient SVD
• O(N) complexity via shift invariance
• Scalable singular value computation