Fast Structured Orthogonal Dictionary Learning using Householder Reflections

2024-09-13

arXiv.org

AI Summary

This work addresses structured orthogonal dictionary learning, specifically for dictionaries parameterized as a single Householder matrix or a product of multiple Householder matrices. We propose an efficient learning algorithm grounded in Householder reflections, which explicitly enforces the structured orthogonality constraint. The algorithm achieves optimal theoretical computational complexity—O(nd)—while attaining high recovery accuracy. We establish, for the first time, approximate recoverability of Householder dictionaries under the ℓ∞ norm and extend this guarantee to multi-reflection products. Through sample complexity analysis and derivation of ℓ∞ error bounds, we prove that our method matches or surpasses state-of-the-art approaches in estimation accuracy—especially in low-sample regimes—while substantially reducing computational cost. Extensive experiments corroborate both the theoretical guarantees and superior practical performance.

Technology Category

• Machine Learning: Structured Learning
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

In this paper, we propose and investigate algorithms for the structured orthogonal dictionary learning problem. First, we investigate the case when the dictionary is a Householder matrix. We give sample complexity results and show theoretically guaranteed approximate recovery (in the $l_{infty}$ sense) with optimal computational complexity. We then attempt to generalize these techniques when the dictionary is a product of a few Householder matrices. We numerically validate these techniques in the sample-limited setting to show performance similar to or better than existing techniques while having much improved computational complexity.

Problem

• Structured orthogonal dictionary learning using Householder reflections
• Sample complexity and approximate recovery guarantees
• Improved computational complexity over existing techniques

Innovation

• Householder matrix dictionary learning
• Approximate recovery with optimal complexity
• Product of Householder matrices generalization