Simultaneous Approximation of the Score Function and Its Derivatives by Deep Neural Networks

2025-12-29

AI Summary

This work studies the simultaneous approximation of score functions and their arbitrary-order derivatives by deep neural networks, focusing on low-dimensional structured probability distributions with unbounded support. Addressing limitations of conventional approaches—which rely on bounded support assumptions and struggle with high-order derivatives—we establish, for the first time, a dimension-free uniform approximation theory for high-order derivatives, removing the bounded-support requirement. Methodologically, we integrate the nonlinear approximation capacity of deep ReLU networks, regularity analysis of density functions, and embedding theorems for high-order Sobolev spaces. Theoretically, we prove that the resulting approximation error achieves the current optimal rate, with convergence strictly independent of input dimension. This provides a rigorous theoretical foundation—under unbounded support—for score-based generative modeling and Bayesian inference.

Technology Category

• Machine Learning: Learning with Manifolds
• Reasoning under Uncertainty: Relational Probabilistic Models
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We present a theory for simultaneous approximation of the score function and its derivatives, enabling the handling of data distributions with low-dimensional structure and unbounded support. Our approximation error bounds match those in the literature while relying on assumptions that relax the usual bounded support requirement. Crucially, our bounds are free from the curse of dimensionality. Moreover, we establish approximation guarantees for derivatives of any prescribed order, extending beyond the commonly considered first-order setting.

Problem

• Approximates score function and derivatives simultaneously
• Handles low-dimensional data with unbounded support
• Avoids dimensionality curse for any derivative order

Innovation

• Simultaneous approximation of score function and derivatives
• Relaxes bounded support requirement for distributions
• Avoids curse of dimensionality in approximation bounds