On best approximation by multivariate ridge functions with applications to generalized translation networks

2024-12-11

arXiv.org

AI Summary

This paper investigates the optimal approximation of $d$-dimensional Sobolev functions in $W^{r,p}([0,1]^d)$ by multivariate ridge functions of the form $sum_k h_k(A_k x)$, where each $A_k$ is a rank-$ell$ matrix. Methodologically, it integrates tools from Sobolev space theory, nonlinear approximation analysis, matrix projection, and harmonic analysis. The main contributions are: (i) the first derivation of asymptotically tight upper and lower bounds—of order $n^{-r(d-ell)/d}$—on the $L^p$-approximation error; the lower bound holds for $L^infty$-Sobolev functions under $L^1$-error, while the upper bound applies for all $1 leq p leq infty$; (ii) the first extension of ridge function approximation theory to generalized translation networks and complex-valued neural networks, yielding optimal approximation rates for Sobolev functions. These results unify and generalize classical univariate ridge function theory, establishing a new theoretical benchmark for high-dimensional function approximation and neural network expressivity.

Technology Category

• Machine Learning: Learning with Manifolds
• Search and Optimization: Non-convex Optimization
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We prove sharp upper and lower bounds for the approximation of Sobolev functions by sums of multivariate ridge functions, i.e., functions of the form $mathbb{R}^d i x mapsto sum_{k=1}^n h_k(A_k x) in mathbb{R}$ with $h_k : mathbb{R}^ell o mathbb{R}$ and $A_k in mathbb{R}^{ell imes d}$. We show that the order of approximation asymptotically behaves as $n^{-r/(d-ell)}$, where $r$ is the regularity of the Sobolev functions to be approximated. Our lower bound even holds when approximating $L^infty$-Sobolev functions of regularity $r$ with error measured in $L^1$, while our upper bound applies to the approximation of $L^p$-Sobolev functions in $L^p$ for any $1 leq p leq infty$. These bounds generalize well-known results about the approximation properties of univariate ridge functions to the multivariate case. Moreover, we use these bounds to obtain sharp asymptotic bounds for the approximation of Sobolev functions using generalized translation networks and complex-valued neural networks.

Problem

• Sharp bounds for approximating Sobolev functions with multivariate ridge functions
• Generalizing univariate ridge function results to multivariate cases
• Applying bounds to generalized translation and complex-valued neural networks

Innovation

• Multivariate ridge functions approximation bounds
• Generalized translation networks application
• Sharp asymptotic bounds for Sobolev functions