Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks

2026-01-29

AI Summary

This work addresses the challenge of approximating the target conditional density in the reverse process of diffusion models, where the density takes the form of a ratio of two kernel densities. To tackle this problem, we propose a deep neural network framework based on the SignReLU activation function. We are the first to employ SignReLU networks for ratio-type function approximation, establishing theoretical bounds on the approximation error and convergence rates in $L^p$ spaces. Furthermore, we decompose the Kullback–Leibler (KL) risk into approximation and estimation errors, and provide a finite-sample upper bound on the KL risk for the reverse process estimator. This analysis yields a generalization guarantee for diffusion models and demonstrates the effectiveness and superiority of SignReLU networks in approximating kernel density ratios.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Motivated by challenges in conditional generative modeling, where the target conditional density takes the form of a ratio f1 over f2, this paper develops a theoretical framework for approximating such ratio-type functionals. Here, f1 and f2 are kernel-based marginal densities that capture structured interactions, a setting central to diffusion-based generative models. We provide a concise proof for approximating these ratio-type functionals using deep neural networks with the SignReLU activation function, leveraging the activation's piecewise structure. Under standard regularity assumptions, we establish L^p(Omega) approximation bounds and convergence rates. Specializing to Denoising Diffusion Probabilistic Models (DDPMs), we construct a SignReLU-based neural estimator for the reverse process and derive bounds on the excess Kullback-Leibler (KL) risk between the generated and true data distributions. Our analysis decomposes this excess risk into approximation and estimation error components. These results provide generalization guarantees for finite-sample training of diffusion-based generative models.

Problem

• ratio-based function approximation
• conditional generative modeling
• diffusion models
• density ratio estimation
• function approximation

Innovation

• SignReLU networks
• ratio-based function approximation
• diffusion models
• KL risk decomposition
• L^p approximation bounds