Calibration Prediction Interval for Non-parametric Regression and Neural Networks

2025-09-02

AI Summary

Traditional prediction intervals (PIs) in regression analysis rely on restrictive linearity and normality assumptions, often violated in practice. To address this, we propose calibrated prediction intervals (cPIs), a distribution-free framework applicable to flexible nonparametric estimators—including kernel methods and deep neural networks—without requiring parametric or distributional assumptions. cPIs explicitly model estimation uncertainty and incorporate a data-driven calibration procedure to guarantee asymptotically valid coverage while maintaining robustness in finite samples. We establish theoretical coverage consistency for a broad class of estimators, including both kernel-based and neural network regressors. Empirical evaluations on synthetic and real-world datasets demonstrate that cPIs achieve near-nominal coverage across diverse settings, significantly outperforming existing nonparametric PI methods—particularly under limited sample sizes—while preserving computational tractability and interpretability.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Probabilistic Programming
• Intelligent Robots: State Estimation

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks

Abstract

Accurate conditional prediction in the regression setting plays an important role in many real-world problems. Typically, a point prediction often falls short since no attempt is made to quantify the prediction accuracy. Classically, under the normality and linearity assumptions, the Prediction Interval (PI) for the response variable can be determined routinely based on the $t$ distribution. Unfortunately, these two assumptions are rarely met in practice. To fully avoid these two conditions, we develop a so-called calibration PI (cPI) which leverages estimations by Deep Neural Networks (DNN) or kernel methods. Moreover, the cPI can be easily adjusted to capture the estimation variability within the prediction procedure, which is a crucial error source often ignored in practice. Under regular assumptions, we verify that our cPI has an asymptotically valid coverage rate. We also demonstrate that cPI based on the kernel method ensures a coverage rate with a high probability when the sample size is large. Besides, with several conditions, the cPI based on DNN works even with finite samples. A comprehensive simulation study supports the usefulness of cPI, and the convincing performance of cPI with a short sample is confirmed with two empirical datasets.

Problem

• Develops calibration prediction intervals for non-parametric regression
• Addresses limitations of normality and linearity assumptions
• Ensures valid coverage rates for neural networks and kernel methods

Innovation

• Uses calibration prediction intervals for non-parametric regression
• Leverages deep neural networks and kernel methods
• Ensures asymptotically valid coverage rate under assumptions