Robust Predictive Uncertainty and Double Descent in Contaminated Bayesian Random Features

2026-02-22

AI Summary

This work addresses the lack of robustness in predictive uncertainty quantification of Bayesian random feature models under misspecified priors and likelihoods. To this end, the authors propose a pessimistic generalized Bayesian inference framework based on a Huber-type contamination set. By explicitly modeling uncertainty in both the prior and the likelihood, the approach constructs a posterior predictive distribution that is robust to misspecification and introduces the Imprecise Highest Density Region (IHDR) to characterize worst-case credible sets. Theoretical analysis shows that the resulting computable upper and lower bounds on the predictive density not only preserve the classical double-descent phase transition structure but also provide rigorous worst-case guarantees. Furthermore, an efficient approximation algorithm for the IHDR enables scalable and robust uncertainty quantification.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Intelligent Robots: State Estimation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

We propose a robust Bayesian formulation of random feature (RF) regression that accounts explicitly for prior and likelihood misspecification via Huber-style contamination sets. Starting from the classical equivalence between ridge-regularized RF training and Bayesian inference with Gaussian priors and likelihoods, we replace the single prior and likelihood with $ε$- and $η$-contaminated credal sets, respectively, and perform inference using pessimistic generalized Bayesian updating. We derive explicit and tractable bounds for the resulting lower and upper posterior predictive densities. These bounds show that, when contamination is moderate, prior and likelihood ambiguity effectively acts as a direct contamination of the posterior predictive distribution, yielding uncertainty envelopes around the classical Gaussian predictive. We introduce an Imprecise Highest Density Region (IHDR) for robust predictive uncertainty quantification and show that it admits an efficient outer approximation via an adjusted Gaussian credible interval. We further obtain predictive variance bounds (under a mild truncation approximation for the upper bound) and prove that they preserve the leading-order proportional-growth asymptotics known for RF models. Together, these results establish a robustness theory for Bayesian random features: predictive uncertainty remains computationally tractable, inherits the classical double-descent phase structure, and is improved by explicit worst-case guarantees under bounded prior and likelihood misspecification.

Problem

• predictive uncertainty
• Bayesian random features
• model misspecification
• robustness
• double descent

Innovation

• robust Bayesian inference
• random features
• predictive uncertainty
• double descent
• imprecise probability