Fundamental Bias in Inverting Random Sampling Matrices with Application to Sub-sampled Newton

2025-02-19

arXiv.org

AI Summary

This work addresses a systematic bias in the inversion of randomized sketching matrices—specifically, the inverse of an unbiased sketching matrix is not necessarily unbiased—a deficiency that severely impairs the convergence and stability of stochastic optimization methods such as subsampled Newton. We propose the first exact inverse bias correction framework applicable to arbitrary sampling mechanisms, including uniform sampling, leverage-score-based sampling, and structured Hadamard-type projections, thereby overcoming prior theoretical limitations restricted to Gaussian or sparse sub-Gaussian projections. We establish a problem-agnostic local convergence rate theory and rigorously prove that the corrected subsampled Newton method achieves superlinear convergence. Empirical evaluations demonstrate substantial improvements in both convergence speed and robustness across large-scale optimization tasks.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Non-convex Optimization
• Machine Learning: Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

A substantial body of work in machine learning (ML) and randomized numerical linear algebra (RandNLA) has exploited various sorts of random sketching methodologies, including random sampling and random projection, with much of the analysis using Johnson--Lindenstrauss and subspace embedding techniques. Recent studies have identified the issue of inversion bias -- the phenomenon that inverses of random sketches are not unbiased, despite the unbiasedness of the sketches themselves. This bias presents challenges for the use of random sketches in various ML pipelines, such as fast stochastic optimization, scalable statistical estimators, and distributed optimization. In the context of random projection, the inversion bias can be easily corrected for dense Gaussian projections (which are, however, too expensive for many applications). Recent work has shown how the inversion bias can be corrected for sparse sub-gaussian projections. In this paper, we show how the inversion bias can be corrected for random sampling methods, both uniform and non-uniform leverage-based, as well as for structured random projections, including those based on the Hadamard transform. Using these results, we establish problem-independent local convergence rates for sub-sampled Newton methods.

Problem

• Inversion bias in random sketches hinders ML pipelines
• Correcting bias in random sampling and structured projections
• Improving convergence rates for sub-sampled Newton methods

Innovation

• Correct inversion bias in random sampling
• Address bias in structured random projections
• Establish local convergence for sub-sampled Newton