ReLU Networks as Random Functions: Their Distribution in Probability Space

2025-03-28

AI Summary

This paper investigates the probabilistic behavior of post-trained ReLU networks under random inputs, modeling them as piecewise affine functions whose structure stochastically switches according to the input distribution. Methodologically, it explicitly derives the discrete distribution over affine functions induced by ReLU activation patterns, establishes an analytical connection between the network’s output distribution and Gaussian orthant probabilities, and proposes a computationally efficient support-set-based approximation algorithm. The approach integrates techniques from piecewise linear system modeling, stochastic activation pattern inference, and numerical approximation. Key contributions include: (1) closed-form, computable expressions for both the activation pattern distribution and the output distribution; (2) characterization of the structured stochasticity inherent in ReLU network outputs—revealing nontrivial dependence on input geometry and weight configuration; and (3) a theoretically grounded framework enabling practical uncertainty quantification and reliability assessment for deep ReLU networks.

Technology Category

• Reasoning under Uncertainty: Relational Probabilistic Models
• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

This paper presents a novel framework for understanding trained ReLU networks as random, affine functions, where the randomness is induced by the distribution over the inputs. By characterizing the probability distribution of the network's activation patterns, we derive the discrete probability distribution over the affine functions realizable by the network. We extend this analysis to describe the probability distribution of the network's outputs. Our approach provides explicit, numerically tractable expressions for these distributions in terms of Gaussian orthant probabilities. Additionally, we develop approximation techniques to identify the support of affine functions a trained ReLU network can realize for a given distribution of inputs. Our work provides a framework for understanding the behavior and performance of ReLU networks corresponding to stochastic inputs, paving the way for more interpretable and reliable models.

Problem

• Characterize probability distribution of ReLU network activation patterns
• Derive discrete distribution over affine functions realizable by network
• Develop approximation techniques for identifying affine function support

Innovation

• Characterizes ReLU networks as random affine functions
• Uses Gaussian orthant probabilities for distributions
• Develops approximation techniques for function support