Improving the Linearized Laplace Approximation via Quadratic Approximations

2026-02-03

AI Summary

Deep neural networks often exhibit overconfidence in out-of-distribution predictions, necessitating efficient and accurate Bayesian uncertainty quantification. This work proposes the Quadratic Laplace Approximation (QLA), which retains the low computational overhead of the Linearized Laplace Approximation (LLA) while more faithfully approximating the full Laplace approximation by capturing second-order information of the log-posterior. Specifically, QLA efficiently constructs a rank-one approximation via power iteration to model curvature beyond the linearization. Experiments across five regression benchmark datasets demonstrate that QLA consistently yields modest but stable improvements over LLA in uncertainty estimation, confirming its effectiveness and practical utility for scalable Bayesian inference in deep learning.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Deep neural networks (DNNs) often produce overconfident out-of-distribution predictions, motivating Bayesian uncertainty quantification. The Linearized Laplace Approximation (LLA) achieves this by linearizing the DNN and applying Laplace inference to the resulting model. Importantly, the linear model is also used for prediction. We argue this linearization in the posterior may degrade fidelity to the true Laplace approximation. To alleviate this problem, without increasing significantly the computational cost, we propose the Quadratic Laplace Approximation (QLA). QLA approximates each second order factor in the approximate Laplace log-posterior using a rank-one factor obtained via efficient power iterations. QLA is expected to yield a posterior precision closer to that of the full Laplace without forming the full Hessian, which is typically intractable. For prediction, QLA also uses the linearized model. Empirically, QLA yields modest yet consistent uncertainty estimation improvements over LLA on five regression datasets.

Problem

• uncertainty quantification
• Laplace approximation
• deep neural networks
• out-of-distribution prediction
• posterior approximation

Innovation

• Quadratic Laplace Approximation
• Bayesian uncertainty quantification
• Linearized Laplace Approximation
• power iterations
• deep neural networks