Fast Swap-Based Element Selection for Multiplication-Free Dimension Reduction

2026-02-14

AI Summary

This work proposes an efficient, multiplication-free dimensionality reduction method tailored for resource-constrained systems, where traditional approaches like PCA are impractical due to their reliance on computationally expensive matrix multiplications. The method achieves dimensionality reduction by selecting a subset of input vector coordinates, with subset quality evaluated via either the minimum mean squared error in linear regression or reconstruction error. The key innovation lies in the novel integration of swap-based local search with the matrix inversion lemma, enabling an efficient solution to this combinatorial optimization problem while significantly reducing computational complexity. Experiments on the MNIST dataset demonstrate that the proposed approach maintains competitive dimensionality reduction performance despite completely eliminating multiplication operations.

Technology Category

• Machine Learning: Dimensionality Reduction/Feature Selection
• Search and Optimization: Non-convex Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

In this paper, we propose a fast algorithm for element selection, a multiplication-free form of dimension reduction that produces a dimension-reduced vector by simply selecting a subset of elements from the input. Dimension reduction is a fundamental technique for reducing unnecessary model parameters, mitigating overfitting, and accelerating training and inference. A standard approach is principal component analysis (PCA), but PCA relies on matrix multiplications; on resource-constrained systems, the multiplication count itself can become a bottleneck. Element selection eliminates this cost because the reduction consists only of selecting elements, and thus the key challenge is to determine which elements should be retained. We evaluate a candidate subset through the minimum mean-squared error of linear regression that predicts a target vector from the selected elements, where the target may be, for example, a one-hot label vector in classification. When an explicit target is unavailable, the input itself can be used as the target, yielding a reconstruction-based criterion. The resulting optimization is combinatorial, and exhaustive search is impractical. To address this, we derive an efficient formula for the objective change caused by swapping a selected and an unselected element, using the matrix inversion lemma, and we perform a swap-based local search that repeatedly applies objective-decreasing swaps until no further improvement is possible. Experiments on MNIST handwritten-digit images demonstrate the effectiveness of the proposed method.

Problem

• element selection
• multiplication-free dimension reduction
• combinatorial optimization
• dimension reduction
• subset selection

Innovation

• multiplication-free dimension reduction
• element selection
• swap-based optimization
• matrix inversion lemma
• combinatorial optimization