Confidence Interval Construction and Conditional Variance Estimation with Dense ReLU Networks

2024-12-29

AI Summary

This paper addresses conditional variance estimation and confidence interval construction in nonparametric regression. Methodologically, it proposes the first non-asymptotic framework for conditional variance estimation using ReLU dense networks, jointly modeling the conditional mean and variance via residual analysis, relaxing the noise assumption to sub-Exponential distributions, and designing a robust bootstrap algorithm with theoretical coverage guarantees. Theoretical contributions include: (1) the first tight non-asymptotic error bound for conditional variance estimation by ReLU networks; (2) attainment of optimal convergence rates under both heteroscedastic and homoscedastic settings; and (3) construction of confidence intervals with rigorous finite-sample coverage guarantees. Empirical results demonstrate that the method significantly improves the reliability and robustness of uncertainty quantification in deep learning models.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Intelligent Robots: State Estimation

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements

Abstract

This paper addresses the problems of conditional variance estimation and confidence interval construction in nonparametric regression using dense networks with the Rectified Linear Unit (ReLU) activation function. We present a residual-based framework for conditional variance estimation, deriving nonasymptotic bounds for variance estimation under both heteroscedastic and homoscedastic settings. We relax the sub-Gaussian noise assumption, allowing the proposed bounds to accommodate sub-Exponential noise and beyond. Building on this, for a ReLU neural network estimator, we derive non-asymptotic bounds for both its conditional mean and variance estimation, representing the first result for variance estimation using ReLU networks. Furthermore, we develop a ReLU network based robust bootstrap procedure (Efron, 1992) for constructing confidence intervals for the true mean that comes with a theoretical guarantee on the coverage, providing a significant advancement in uncertainty quantification and the construction of reliable confidence intervals in deep learning settings.

Problem

• Uncertainty Estimation
• ReLU Networks
• Conditional Variance

Innovation

• ReLU Networks
• Uncertainty Estimation
• Confidence Intervals