Harnessing Multiple Large Language Models: A Survey on LLM Ensemble

2025-02-25

AI Summary

This study addresses the suboptimal collaborative reasoning performance among multiple large language models (LLMs). To tackle this, we propose the first comprehensive LLM ensemble taxonomy spanning the *pre-reasoning*, *in-reasoning*, and *post-reasoning* stages. Through a systematic literature review and cross-method comparative analysis, we rigorously define three core ensemble paradigms and establish a unified classification framework. Concurrently, we release an open-source GitHub repository—including a curated collection of methods, standardized benchmark evaluations, and real-world application cases—constituting the first systematic survey on LLM ensembling. Our work bridges a critical gap in the field by providing the first structured theoretical foundation for ensemble techniques. Moreover, it delivers a reusable research roadmap and empirical basis for method evaluation, tool development, and practical deployment—thereby advancing LLM ensembling from ad hoc practice toward rigorous, systematized research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

LLM Ensemble -- which involves the comprehensive use of multiple large language models (LLMs), each aimed at handling user queries during downstream inference, to benefit from their individual strengths -- has gained substantial attention recently. The widespread availability of LLMs, coupled with their varying strengths and out-of-the-box usability, has profoundly advanced the field of LLM Ensemble. This paper presents the first systematic review of recent developments in LLM Ensemble. First, we introduce our taxonomy of LLM Ensemble and discuss several related research problems. Then, we provide a more in-depth classification of the methods under the broad categories of"ensemble-before-inference, ensemble-during-inference, ensemble-after-inference", and review all relevant methods. Finally, we introduce related benchmarks and applications, summarize existing studies, and suggest several future research directions. A curated list of papers on LLM Ensemble is available at https://github.com/junchenzhi/Awesome-LLM-Ensemble.

Problem

• Survey on LLM Ensemble methods
• Classification of ensemble strategies
• Future research directions in LLM

Innovation

• Combines multiple LLMs effectively
• Classifies ensemble methods systematically
• Introduces benchmarks for LLM Ensemble