The Bernstein-von Mises theorem for Bayesian one-pass online learning

2026-04-30

AI Summary

This work addresses the lack of theoretical guarantees in single-pass Bayesian online learning, where existing methods rely on asymptotic assumptions involving diverging mini-batch sizes and thus fail to reflect practical streaming settings. To overcome this limitation, the authors propose a novel algorithm incorporating a warm-start mechanism to stabilize sequential posterior updates. They establish, for the first time, a Bernstein–von Mises theorem under a genuine single-pass regime without requiring diverging batch sizes, demonstrating that the resulting posterior achieves optimal contraction rates and provides reliable uncertainty quantification. The study develops a new theoretical framework and validates the method on generalized linear models, showing that its performance matches that of batch estimators and substantially outperforms existing online algorithms.

Abstract

Bayesian online learning provides a coherent framework for sequential inference. However, its theoretical understanding remains limited, particularly in the one-pass setting. Existing theoretical guarantees typically require the mini-batch sample size to diverge, a condition that fails in the one-pass regime. In this paper, we propose a new Bayesian online learning algorithm tailored to the one-pass setting, which incorporates a warm-start phase to ensure stable sequential updates. For this algorithm, we show that the sequentially updated posterior attains the optimal convergence rate. Building on this, we establish an online analogue of the Bernstein-von Mises theorem, which guarantees valid uncertainty quantification without diverging mini-batch sample sizes. Our analysis is based on a novel theoretical framework that differs fundamentally from existing approaches in the online learning literature. Numerical experiments on generalized linear models show that the proposed method matches the performance of the batch estimator while outperforming existing online procedures.

Problem

• Bayesian online learning
• one-pass setting
• Bernstein-von Mises theorem
• uncertainty quantification
• sequential inference

Innovation

• Bayesian online learning
• one-pass setting
• Bernstein-von Mises theorem
• warm-start
• uncertainty quantification