Learning Lipschitz Operators with respect to Gaussian Measures with Near-Optimal Sample Complexity

2024-10-30

arXiv.org

AI Summary

This work addresses the weak theoretical foundation for learning Lipschitz operators in infinite-dimensional spaces. We study efficient approximation from finite samples under Gaussian measure. We propose a Christoffel-weighted least-squares algorithm that integrates adaptive linear sampling with Hermite polynomial expansions, and establish tight, matching upper and lower bounds on the approximation error within a Gaussian Sobolev regularity framework. Our analysis yields the first proof of optimality for Hermite approximations under adaptive sampling. We reveal that the spectral decay rate fundamentally governs the convergence order: when the spectrum decays sufficiently fast, the convergence rate can approach an arbitrarily high algebraic rate. Moreover, the algorithm achieves near-optimal sample complexity with high probability, thereby substantially strengthening both the theoretical guarantees and computational efficiency of operator learning.

Technology Category

• Machine Learning: Learning with Manifolds
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Operator learning, the approximation of mappings between infinite-dimensional function spaces using ideas from machine learning, has gained increasing research attention in recent years. Approximate operators, learned from data, hold promise to serve as efficient surrogate models for problems in computational science and engineering, complementing traditional numerical methods. However, despite their empirical success, our understanding of the underpinning mathematical theory is in large part still incomplete. In this paper, we study the approximation of Lipschitz operators in expectation with respect to Gaussian measures. We prove higher Gaussian Sobolev regularity of Lipschitz operators and establish lower and upper bounds on the Hermite polynomial approximation error. We further consider the reconstruction of Lipschitz operators from $m$ arbitrary (adaptive) linear samples. A key finding is the tight characterization of the smallest achievable error for all possible (adaptive) sampling and reconstruction maps in terms of $m$. It is shown that Hermite polynomial approximation is an optimal recovery strategy, but we have the following curse of sample complexity: No method to approximate Lipschitz operators based on $m$ samples can achieve algebraic convergence rates in $m$. On the positive side, we prove that a sufficiently fast spectral decay of the covariance operator of the Gaussian measure guarantees convergence rates which are arbitrarily close to any algebraic rate in the large data limit $m o infty$. A main focus of this work is on the recovery of Lipschitz operators from finitely many point samples. We use Christoffel sampling and weighted least-squares approximation to propose an algorithm which provably achieves near-optimal sample complexity in high probability.

Problem

• Gaussian Measures
• Lipschitz Operators
• Operator Learning

Innovation

• Lipschitz operators approximation
• Gaussian measures
• Hermite polynomial methods