Optimal sampling for least-squares approximation

2024-09-04

arXiv.org

AI Summary

This work addresses the design of **optimal randomized sampling strategies** for weighted least-squares approximation, extending beyond classical settings restricted to pointwise evaluations and linear approximation spaces to encompass **generalized non-pointwise observations** (e.g., integrals, derivatives) and **nonlinear approximation spaces**. Methodologically, it introduces a systematic generalization of the Christoffel function to the **generalized recovery framework**, yielding a unified theoretical foundation. The resulting sampling scheme achieves near-optimal sample complexity—requiring only $O(n log n)$ measurements for $n$ degrees of freedom—improving upon the classical $O(n^2)$ bound. Theoretical analysis guarantees stable, high-probability reconstruction. The approach integrates tools from approximation theory, randomized sampling design, and numerical linear algebra. Extensive experiments demonstrate its effectiveness and broad applicability in machine learning and scientific computing.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Learning with Manifolds

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Least-squares approximation is one of the most important methods for recovering an unknown function from data. While in many applications the data is fixed, in many others there is substantial freedom to choose where to sample. In this paper, we review recent progress on optimal sampling for (weighted) least-squares approximation in arbitrary linear spaces. We introduce the Christoffel function as a key quantity in the analysis of (weighted) least-squares approximation from random samples, then show how it can be used to construct sampling strategies that possess near-optimal sample complexity: namely, the number of samples scales log-linearly in $n$, the dimension of the approximation space. We discuss a series of variations, extensions and further topics, and throughout highlight connections to approximation theory, machine learning, information-based complexity and numerical linear algebra. Finally, motivated by various contemporary applications, we consider a generalization of the classical setting where the samples need not be pointwise samples of a scalar-valued function, and the approximation space need not be linear. We show that even in this significantly more general setting suitable generalizations of the Christoffel function still determine the sample complexity. This provides a unified procedure for designing improved sampling strategies for general recovery problems. This article is largely self-contained, and intended to be accessible to nonspecialists.

Problem

• Designing near-optimal random sampling for least-squares approximation
• Using Christoffel function to determine sample complexity
• Extending sampling strategies to non-linear and non-pointwise settings

Innovation

• Uses Christoffel function for optimal sampling
• Introduces near-optimal Christoffel sampling strategy
• Generalizes sampling for nonlinear approximation spaces