Activation-Space Uncertainty Quantification for Pretrained Networks

2026-02-16

AI Summary

This work addresses the challenge of efficiently and reliably estimating epistemic uncertainty in frozen pre-trained models. The authors propose GAPA, a method that constructs a Gaussian process model in the activation space, leveraging a replaceable nonlinear activation function to enable closed-form quantification of epistemic uncertainty. By shifting Bayesian modeling from weight space to activation space, GAPA requires neither retraining, sampling, nor backpropagation, and yields uncertainty estimates with a single forward pass. Integrating sparse variational inducing points, local k-nearest-neighbor conditioning, and cached training activations, GAPA achieves competitive or superior performance compared to existing post-hoc baselines in out-of-distribution detection and calibration across diverse tasks—including regression, classification, image segmentation, and language modeling—while maintaining high efficiency at test time.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Reliable uncertainty estimates are crucial for deploying pretrained models; yet, many strong methods for quantifying uncertainty require retraining, Monte Carlo sampling, or expensive second-order computations and may alter a frozen backbone's predictions. To address this, we introduce Gaussian Process Activations (GAPA), a post-hoc method that shifts Bayesian modeling from weights to activations. GAPA replaces standard nonlinearities with Gaussian-process activations whose posterior mean exactly matches the original activation, preserving the backbone's point predictions by construction while providing closed-form epistemic variances in activation space. To scale to modern architectures, we use a sparse variational inducing-point approximation over cached training activations, combined with local k-nearest-neighbor subset conditioning, enabling deterministic single-pass uncertainty propagation without sampling, backpropagation, or second-order information. Across regression, classification, image segmentation, and language modeling, GAPA matches or outperforms strong post-hoc baselines in calibration and out-of-distribution detection while remaining efficient at test time.

Problem

• uncertainty quantification
• pretrained networks
• post-hoc calibration
• epistemic uncertainty
• frozen backbone

Innovation

• Gaussian Process Activations
• activation-space uncertainty
• post-hoc uncertainty quantification
• epistemic uncertainty
• sparse variational approximation