Ensembling Language Models with Sequential Monte Carlo

2026-03-05

AI Summary

This work addresses the challenge that existing language model ensemble methods struggle to accurately approximate the joint distribution over the full string space due to local normalization bias during decoding. To overcome this limitation, the authors propose a unified f-ensemble framework that, for the first time, integrates Sequential Monte Carlo (SMC) into language model ensembling. By operating in a byte-level shared character space, the method enables unbiased, globally consistent sampling across heterogeneous vocabularies and supports flexible combination of multiple models under arbitrary aggregation functions \( f \). Experimental results demonstrate that the proposed approach significantly outperforms conventional probability averaging strategies across various structured text generation tasks, highlighting the critical role of improved posterior approximation in enhancing ensemble performance.

Technology Category

• Machine Learning: Ensemble Methods
• Natural Language Processing: (Large) Language Models
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Practitioners have access to an abundance of language models and prompting strategies for solving many language modeling tasks; yet prior work shows that modeling performance is highly sensitive to both choices. Classical machine learning ensembling techniques offer a principled approach: aggregate predictions from multiple sources to achieve better performance than any single one. However, applying ensembling to language models during decoding is challenging: naively aggregating next-token probabilities yields samples from a locally normalized, biased approximation of the generally intractable ensemble distribution over strings. In this work, we introduce a unified framework for composing $K$ language models into $f$-ensemble distributions for a wide range of functions $f\colon\mathbb{R}_{\geq 0}^{K}\to\mathbb{R}_{\geq 0}$. To sample from these distributions, we propose a byte-level sequential Monte Carlo (SMC) algorithm that operates in a shared character space, enabling ensembles of models with mismatching vocabularies and consistent sampling in the limit. We evaluate a family of $f$-ensembles across prompt and model combinations for various structured text generation tasks, highlighting the benefits of alternative aggregation strategies over traditional probability averaging, and showing that better posterior approximations can yield better ensemble performance.

Problem

• language model ensembling
• sequential Monte Carlo
• text generation
• decoding
• ensemble distribution

Innovation

• f-ensembles
• Sequential Monte Carlo
• language model ensembling
• byte-level sampling
• structured text generation