Sampling by averaging: A multiscale approach to score estimation

2025-08-20

AI Summary

To address efficient sampling from complex, unnormalized target distributions, this paper proposes a novel framework grounded in the stochastic averaging principle for multiscale stochastic differential equation (SDE) slow–fast systems. The method directly estimates intermediate scores along diffusion trajectories via dynamical averaging—eliminating the need for explicit score-model training or nested MCMC loops. This work constitutes the first application of stochastic averaging theory to score estimation, offering rigorous theoretical convergence guarantees and intrinsic robustness to heavy-tailed noise (e.g., Student’s *t* noise). By integrating annealed Langevin dynamics with controlled diffusion processes, the resulting sampler achieves accuracy and efficiency comparable to state-of-the-art methods on multimodal and high-dimensional benchmarks, while requiring zero training.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Security and Privacy: Large-scale security measurements

Abstract

We introduce a novel framework for efficient sampling from complex, unnormalised target distributions by exploiting multiscale dynamics. Traditional score-based sampling methods either rely on learned approximations of the score function or involve computationally expensive nested Markov chain Monte Carlo (MCMC) loops. In contrast, the proposed approach leverages stochastic averaging within a slow-fast system of stochastic differential equations (SDEs) to estimate intermediate scores along a diffusion path without training or inner-loop MCMC. Two algorithms are developed under this framework: MultALMC, which uses multiscale annealed Langevin dynamics, and MultCDiff, based on multiscale controlled diffusions for the reverse-time Ornstein-Uhlenbeck process. Both overdamped and underdamped variants are considered, with theoretical guarantees of convergence to the desired diffusion path. The framework is extended to handle heavy-tailed target distributions using Student's t-based noise models and tailored fast-process dynamics. Empirical results across synthetic and real-world benchmarks, including multimodal and high-dimensional distributions, demonstrate that the proposed methods are competitive with existing samplers in terms of accuracy and efficiency, without the need for learned models.

Problem

• Efficient sampling from complex unnormalized distributions
• Avoiding expensive nested MCMC loops and learned approximations
• Handling heavy-tailed and high-dimensional multimodal distributions

Innovation

• Multiscale SDEs for score estimation without training
• Stochastic averaging replaces inner-loop MCMC sampling
• Student's t-noise models handle heavy-tailed distributions