Time-Frequency Analysis for Neural Networks

2025-12-17

AI Summary

This work addresses the quantitative approximation of Sobolev functions $W^{n,r}(Omega)$ by shallow neural networks in weighted modulation spaces $M^{p,q}_m(mathbb{R}^d)$. We propose a novel time-frequency-based approximation framework that integrates weighted modulation space theory with locally supported time-frequency window activation functions. Our approach achieves, for the first time, a dimension-independent approximation rate of $N^{-1/2}$ (where $N$ denotes the number of neurons), with explicitly controlled constants—unifying approximation guarantees across Feichtinger’s algebra, Fourier–Lebesgue spaces, and Barron spaces. Theoretically, our network attains superior Sobolev-norm approximation accuracy compared to ReLU networks. Numerical experiments in one and two dimensions confirm that the observed error decay aligns precisely with the predicted $N^{-1/2}$ rate, demonstrating significantly enhanced global approximation performance on both bounded domains and the entire space $mathbb{R}^d$.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Models for Web evolution
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

We develop a quantitative approximation theory for shallow neural networks using tools from time-frequency analysis. Working in weighted modulation spaces $M^{p,q}_m(mathbf{R}^{d})$, we prove dimension-independent approximation rates in Sobolev norms $W^{n,r}(Ω)$ for networks whose units combine standard activations with localized time-frequency windows. Our main result shows that for $f in M^{p,q}_m(mathbf{R}^{d})$ one can achieve [ |f - f_N|_{W^{n,r}(Ω)} lesssim N^{-1/2},|f|_{M^{p,q}_m(mathbf{R}^{d})}, ] on bounded domains, with explicit control of all constants. We further obtain global approximation theorems on $mathbf{R}^{d}$ using weighted modulation dictionaries, and derive consequences for Feichtinger's algebra, Fourier-Lebesgue spaces, and Barron spaces. Numerical experiments in one and two dimensions confirm that modulation-based networks achieve substantially better Sobolev approximation than standard ReLU networks, consistent with the theoretical estimates.

Problem

• Develops a quantitative approximation theory for shallow neural networks.
• Proves dimension-independent approximation rates in Sobolev norms.
• Demonstrates improved Sobolev approximation over standard ReLU networks.

Innovation

• Time-frequency analysis for shallow neural networks
• Localized time-frequency windows with standard activations
• Modulation-based networks improve Sobolev approximation rates