Online Covariance Matrix Estimation in Sketched Newton Methods

2025-02-10

AI Summary

While asymptotic normality has been established for sketched Newton methods in streaming settings, existing work lacks an online-computable, matrix-decomposition-free estimator for the limiting covariance. Method: We propose the first fully online, batch-agnostic, factorization-free covariance estimator, leveraging random projections and recursive updates. Theoretically, we prove its strong consistency and derive its convergence rate. Contribution/Results: The estimator enables real-time statistical inference without storing historical data or performing SVD/Cholesky decompositions. In regression tasks and CUTEst benchmark experiments, it achieves significantly higher confidence interval coverage and precision than first-order methods (e.g., OGD) with conventional covariance estimators. This work fills a critical theoretical and methodological gap in second-order online inference—namely, the absence of a practical, asymptotically consistent estimator for the limiting covariance under streaming conditions.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Given the ubiquity of streaming data, online algorithms have been widely used for parameter estimation, with second-order methods particularly standing out for their efficiency and robustness. In this paper, we study an online sketched Newton method that leverages a randomized sketching technique to perform an approximate Newton step in each iteration, thereby eliminating the computational bottleneck of second-order methods. While existing studies have established the asymptotic normality of sketched Newton methods, a consistent estimator of the limiting covariance matrix remains an open problem. We propose a fully online covariance matrix estimator that is constructed entirely from the Newton iterates and requires no matrix factorization. Compared to covariance estimators for first-order online methods, our estimator for second-order methods is batch-free. We establish the consistency and convergence rate of our estimator, and coupled with asymptotic normality results, we can then perform online statistical inference for the model parameters based on sketched Newton methods. We also discuss the extension of our estimator to constrained problems, and demonstrate its superior performance on regression problems as well as benchmark problems in the CUTEst set.

Problem

• Online covariance matrix estimation
• Sketched Newton methods
• Statistical inference for parameters

Innovation

• Online sketched Newton method
• Fully online covariance matrix estimator
• Batch-free second-order estimators