LLMBoost: Make Large Language Models Stronger with Boosting

2025-12-26

AI Summary

Existing LLM ensemble methods typically treat constituent models as black boxes, performing only input or output fusion while neglecting intermediate representations and inter-model interactions. To address this, we propose LLMBoost—a boosting-inspired fine-tuning framework for LLM ensembles. Our key contributions are: (1) cross-model attention, which explicitly models interaction among heterogeneous models’ intermediate representations; (2) a chained error-suppression training paradigm that enables hierarchical error correction and knowledge transfer; and (3) layer-wise state pipelined inference, balancing efficiency and performance. We theoretically prove that sequential ensembling guarantees monotonic performance improvement. Experiments demonstrate that LLMBoost consistently improves accuracy on commonsense and arithmetic reasoning benchmarks while reducing inference latency—achieving efficiency comparable to single-model decoding.

Technology Category

• Machine Learning: Ensemble Methods
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Ensemble learning of LLMs has emerged as a promising alternative to enhance performance, but existing approaches typically treat models as black boxes, combining the inputs or final outputs while overlooking the rich internal representations and interactions across models.In this work, we introduce LLMBoost, a novel ensemble fine-tuning framework that breaks this barrier by explicitly leveraging intermediate states of LLMs. Inspired by the boosting paradigm, LLMBoost incorporates three key innovations. First, a cross-model attention mechanism enables successor models to access and fuse hidden states from predecessors, facilitating hierarchical error correction and knowledge transfer. Second, a chain training paradigm progressively fine-tunes connected models with an error-suppression objective, ensuring that each model rectifies the mispredictions of its predecessor with minimal additional computation. Third, a near-parallel inference paradigm design pipelines hidden states across models layer by layer, achieving inference efficiency approaching single-model decoding. We further establish the theoretical foundations of LLMBoost, proving that sequential integration guarantees monotonic improvements under bounded correction assumptions. Extensive experiments on commonsense reasoning and arithmetic reasoning tasks demonstrate that LLMBoost consistently boosts accuracy while reducing inference latency.

Problem

• Enhances LLM performance via ensemble fine-tuning with intermediate states
• Introduces cross-model attention for hierarchical error correction and transfer
• Achieves efficient inference with near-parallel decoding and reduced latency

Innovation

• Cross-model attention for hierarchical error correction
• Chain training with error-suppression objective
• Near-parallel inference for efficient decoding