A new Linear Time Bi-level ℓ1,∞ projection ; Application to the sparsification of auto-encoders neural networks

2024-07-23

arXiv.org

AI Summary

To address the high computational complexity—currently optimal at 𝒪(nm log(nm))—of projecting onto the matrix ℓ₁,∞ norm, this paper proposes the first linear-time 𝒪(nm) two-level ℓ₁,∞ projection algorithm, specifically designed for structured sparsification in neural networks. Methodologically, we (1) establish and rigorously prove a key ℓ₁,∞ norm identity, providing the theoretical foundation for linear-time computation; (2) devise a two-level optimization framework that tightly couples exact projection with structured sparsity regularization; and (3) evaluate the algorithm on autoencoder pruning tasks. Results show a 2.5× speedup over the state-of-the-art method, achieving the highest sparsity rate without sacrificing classification accuracy—thereby jointly optimizing sparsity and generalization performance.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Search and Optimization: Non-convex Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Security and Privacy: Large-scale security measurements

Abstract

The $ell_{1,infty}$ norm is an efficient-structured projection, but the complexity of the best algorithm is, unfortunately, $mathcal{O}ig(n m log(n m)ig)$ for a matrix $n imes m$.\ In this paper, we propose a new bi-level projection method, for which we show that the time complexity for the $ell_{1,infty}$ norm is only $mathcal{O}ig(n m ig)$ for a matrix $n imes m$. Moreover, we provide a new $ell_{1,infty}$ identity with mathematical proof and experimental validation. Experiments show that our bi-level $ell_{1,infty}$ projection is $2.5$ times faster than the actual fastest algorithm and provides the best sparsity while keeping the same accuracy in classification applications.

Problem

• Proposes a faster bi-level projection method for the $ell_{1, infty}$ norm.
• Reduces time complexity to $mathcal{O}(n m)$ for $n imes m$ matrices.
• Improves sparsity and speed in auto-encoder neural networks.

Innovation

• Linear Time Bi-level projection method
• Reduced complexity to O(nm)
• Enhanced sparsity and classification accuracy