Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling

2024-10-03

arXiv.org

AI Summary

Existing LLM ensemble methods often overlook model compatibility and rely on full-vocabulary probability alignment, resulting in low efficiency and high computational overhead. This work identifies model compatibility as a critical determinant of ensemble performance and proposes Union Top-k Ensemble—a novel paradigm that first selects a compatible subset of models and then performs lightweight probability aggregation solely over the union of each model’s top-k predicted tokens, thereby avoiding full-vocabulary alignment. Our approach establishes the first token-level, compatibility-aware ensemble framework that requires no retraining. Evaluated across multiple benchmark tasks, it significantly outperforms state-of-the-art ensemble methods in both accuracy and robustness while reducing computational cost by 30%–50%.

Technology Category

• Machine Learning: Ensemble Methods
• Natural Language Processing: (Large) Language Models
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Large language models (LLMs) exhibit varying strengths and weaknesses across different tasks, prompting recent studies to explore the benefits of ensembling models to leverage their complementary advantages. However, existing LLM ensembling methods often overlook model compatibility and struggle with inefficient alignment of probabilities across the entire vocabulary. In this study, we empirically investigate the factors influencing ensemble performance, identifying model performance, vocabulary size, and response style as key determinants, revealing that compatibility among models is essential for effective ensembling. This analysis leads to the development of a simple yet effective model selection strategy that identifies compatible models. Additionally, we introduce the extsc{Uni}on extsc{T}op-$k$ extsc{E}nsembling ( extsc{UniTE}), a novel approach that efficiently combines models by focusing on the union of the top-k tokens from each model, thereby avoiding the need for full vocabulary alignment and reducing computational overhead. Extensive evaluations across multiple benchmarks demonstrate that extsc{UniTE} significantly enhances performance compared to existing methods, offering a more efficient framework for LLM ensembling.

Problem

• Identify model compatibility for effective ensembling
• Develop efficient model selection strategy
• Introduce UniTE for top-k token union ensembling

Innovation

• Top-k tokens union
• Model compatibility strategy
• Efficient ensembling framework