Efficient uniform approximation using Random Vector Functional Link networks

2023-06-30

International Conference on Sampling Theory and Applications

AI Summary

This work investigates the uniform approximation capability of two-layer ReLU-activated Random Vector Functional Link (RVFL) networks for Lipschitz continuous functions. While prior analyses are confined to asymptotic L²-convergence, we establish the first non-asymptotic uniform (L∞) approximation theory for RVFL networks. Specifically, we prove that when hidden-layer weights are drawn from a Gaussian distribution and the number of hidden nodes grows exponentially with input dimension, ReLU-RVFL networks achieve arbitrary-precision uniform approximation. We derive an explicit, tight lower bound on the required number of hidden nodes—depending only on the Lipschitz constant of the target function, the desired approximation error, and the input dimension. Our analysis integrates tools from probability theory, harmonic analysis, random matrix theory, and covering number estimation. This result transcends the limitations of L²-based frameworks and provides foundational theoretical guarantees for designing lightweight, provably robust neural networks.

Abstract

A Random Vector Functional Link (RVFL) network is a depth-2 neural network with random inner weights and biases. As only the outer weights of such an architecture need to be learned, the learning process boils down to a linear optimization task, allowing one to sidestep the pitfalls of nonconvex optimization problems. In this paper, we prove that an RVFL with ReLU activation functions can approximate Lipschitz continuous functions provided its hidden layer is exponentially wide in the input dimension. Although it has been established before that such approximation can be achieved in L2 sense, we prove it for L∞ approximation error and Gaussian inner weights. To the best of our knowledge, our result is the first of this kind. We give a non-asymptotic lower bound for the number of hidden layer nodes, depending on, among other things, the Lipschitz constant of the target function, the desired accuracy, and the input dimension. Our method of proof is rooted in probability theory and harmonic analysis.

Problem

• RVFL networks approximate Lipschitz functions in L∞ norm
• Nonasymptotic bounds for hidden nodes needed for accuracy
• Linear optimization avoids nonconvex learning pitfalls

Innovation

• RVFL networks use random inner weights
• Linear optimization for outer weights
• Gaussian weights enable L∞ approximation