Function-Space Empirical Bayes Regularisation with Large Vision-Language Model Priors

2026-02-03

AI Summary

This work addresses the challenge in Bayesian deep learning where designing effective priors for high-dimensional data often leads to unreliable uncertainty quantification. To this end, the authors propose the VLM-FS-EB framework, which introduces large vision-language models (VLMs) into function-space Bayesian inference for the first time. Leveraging the semantic understanding capabilities of VLMs, the method generates high-quality contextual points to construct expressive, non-Gaussian, semantics-aware function priors. These priors are regularized through empirical Bayes combined with function-space variational inference. Experimental results demonstrate that the proposed framework significantly improves both predictive performance and the reliability of uncertainty estimates, particularly in out-of-distribution detection and data-scarce scenarios, outperforming multiple established baselines.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Bayesian deep learning (BDL) provides a principled framework for reliable uncertainty quantification by combining deep neural networks with Bayesian inference. A central challenge in BDL lies in the design of informative prior distributions that scale effectively to high-dimensional data. Recent functional variational inference (VI) approaches address this issue by imposing priors directly in function space; however, most existing methods rely on Gaussian process (GP) priors, whose expressiveness and generalisation capabilities become limited in high-dimensional regimes. In this work, we propose VLM-FS-EB, a novel function-space empirical Bayes regularisation framework, leveraging large vision-language models (VLMs) to generates semantically meaningful context points. These synthetic samples are then used VLMs for embeddings to construct expressive functional priors. Furthermore, the proposed method is evaluated against various baselines, and experimental results demonstrate that our method consistently improves predictive performance and yields more reliable uncertainty estimates, particularly in out-of-distribution (OOD) detection tasks and data-scarce regimes.

Problem

• Bayesian deep learning
• function-space prior
• large vision-language models
• uncertainty quantification
• out-of-distribution detection

Innovation

• function-space prior
• empirical Bayes
• vision-language models
• Bayesian deep learning
• uncertainty quantification